Lesson 3.2 rendered RGB, depth and segmentation; this lesson asks what each one measures, precisely enough to compute with. Depth is metric distance along the camera’s optical axis, not along each pixel’s ray. Segmentation labels every pixel with an object id and an object type, and the types include more than geoms. Inverting the projection of Lesson 11.1 turns depth into a point cloud in the world frame, and a point cloud from one view is a sample of the visible surface, not of the object: its centroid is not the object’s centre.
[!established] What the renderer returns With
mujoco.Renderer,enable_depth_rendering()makesrender()return a float32 array of metres, andenable_segmentation_rendering()returns an int32 array of shape (H, W, 2) holding the object id and themjtObjtype of each pixel, with −1 where nothing was hit.update_scenetakes anMjvOption(scene_option) that controls which geom and site groups are drawn, in segmentation as in RGB.
The lab of Lesson 11.1, here in depth mode, also back-projects the red cube. Read the bottom readout: the centroid of the cube’s point cloud minus its true centre, in millimetres.
Each camera’s error points toward that camera. That is the signature of a visible-surface bias, and no amount of resolution removes it.
```lab camera {“dock”: true, “title”: “Depth, segmentation and a point cloud”, “height”: 260, “mode”: “depth”}
## Mathematics
### Depth is along the axis
For a pixel with normalized coordinates $(a, b) = ((u - c_x)/f_x,\ (v - c_y)/f_y)$, a point at depth $z$ on its ray is $p_c = z\,(a, b, 1)$ in the OpenCV camera frame, and its distance from the camera is $z\sqrt{1 + a^2 + b^2}$. Output (1) separates the two readings with the top camera, 1.4 m above the floor: floor pixels at the image corners read 1.4000 m, while their rays are 1.627 to 1.629 m long. MuJoCo's depth is the $z$ coordinate. A pipeline that treats it as ray length inflates distances by up to 16% at the corners of a 45° camera.
### Depth precision
Output (2) renders a plane from 0.3 to 30 m above it and compares every pixel with the true distance. The largest error at each height equals the spacing of float32 numbers at that distance (0.24 µm at 3 m, 1.91 µm at 30 m), with the near clipping plane at 0.01 m or at 0.1 m. In MuJoCo 3.14.0, rendered depth is exact up to the rounding of its float32 output in this test. Real depth sensors are not: their noise and missing data are a modelling choice you have to add (Level 18), because the simulator will not supply them.
### Segmentation contains sites
Output (3): of the front camera's 307 200 pixels, 4 668 are labelled as sites, not geoms: the two translucent placement zones (`left_zone`, `right_zone`) and the tray's centre marker. They are drawn, so they are segmented, and the floor beneath them loses those pixels. Hiding the site groups in the scene options (`option.sitegroup[:] = 0`) returns every pixel to a geom. Objects made of several geoms need one more step: the tray is five visible geoms of one body, so a body-level mask maps each geom id through `model.geom_bodyid`.
### From depth to points
Back-projection inverts Lesson 11.1's projection pixel by pixel:
$$p_c = z\left(\frac{u + 0.5 - c_x}{f_x},\ \frac{v + 0.5 - c_y}{f_y},\ 1\right),\qquad p_w = R_{wc}^{cv}\,p_c + t_{wc},$$
with $R_{wc}^{cv} = $ `cam_xmat` $\cdot\,\mathrm{diag}(1,-1,-1)$ and $t_{wc} = $ `cam_xpos`. Output (4) checks it on the floor: 240 000 to 280 000 floor points per camera, all within 0.000 mm of $z = 0$. The cube is the instructive case. Its points' centroid misses its centre by 14 to 17 mm in every single view at 640 × 480, always toward the camera; fusing the three views still leaves 4.8, −5.2 and 9.8 mm, because the bottom and the faces turned away from all three cameras are never seen. A shape-aware estimate does better: the centre of the fused points' bounding box is within 0.2 mm, because together the views reach the cube's extremes along every axis. At 320 × 240 the per-camera numbers are those of the lab (11.4, −0.9, 8.5 mm for the front camera against the lab's 11.6, −0.8, 8.6), which is a check that the lab's back-projection matches the script's.
> [!recommendation] Estimate objects with a model of what is visible
> Do not use the centroid of an object's points as its position. Use the part of the object you know you see (the top face from above, offset by half the height), fit a model of the shape to the points, or fuse views that together cover the shape. Then measure the estimator's error over many poses, as output (5) does, before you design anything downstream on the assumption that perception is accurate.
### How good is a camera as a position sensor?
Lesson 10.2 asked how much perception error a pick-and-place tolerates, and measured a capture region of 24 mm by 21 mm. Output (5) measures what a camera delivers: the top camera at 640 × 480 looks at the red cube in the 100 poses of Lesson 10.2, and the estimate is the xy centre of the cube's top-face points. With the arm at its home pose, the arm hides more than a fifth of the top face in 34 of the 100 poses; swung aside, in none. Where the face is visible, the error is 0.15 mm on average, 0.55 mm at the 95th percentile and 0.99 mm at worst, a fraction of the 2.35 mm footprint of one pixel because the centroid averages hundreds of them. In this simulation, perception precision is not what limits the pick-and-place; occlusion by the robot is, and so are the effects simulation leaves out (sensor noise, calibration error), which is the subject of Levels 18 and 19.
## Implementation
```io
INPUT: pick_place.xml; a plane under a downward camera
PROCESS: depth against axis distance and ray length; depth error from 0.3 to 30 m; segmentation contents with and without sites; point clouds per camera and fused; the top camera as a cube-position sensor over 100 poses
OUTPUT: printed tables
```python file=examples/l11_2_depth_pointclouds.py “"”Lesson 11.2: depth, segmentation and point clouds: what the images measure.
INPUT pick_place.xml; a test scene (a large plane under a downward camera at a chosen height) PROCESS (1) what depth is: the top camera’s depth at floor pixels in the centre and the corners of the image, against the camera height and the length of each pixel’s ray; (2) depth accuracy against distance: the plane seen from 0.1 to 30 m, with the default near clipping plane and one ten times farther; (3) segmentation: what the front camera’s labels contain (geoms, sites, background), and the same with sites hidden through the scene options; (4) point clouds: depth back-projected to the world from three cameras; floor flatness; the red cube’s points against its true centre, per camera and fused; (5) the top camera as the cube-position sensor of Lesson 10.2: the xy centre of the red cube’s top face over 100 random cube poses, against the truth OUTPUT printed tables
Run: MUJOCO_GL=egl python examples/l11_2_depth_pointclouds.py (or osmesa without a GPU) “””
import math import os
import mujoco import numpy as np
from mjcourse import model_path
W, H = 640, 480 FLIP = np.diag([1.0, -1.0, -1.0]) # MuJoCo camera axes -> OpenCV (x right, y down, z forward) STATES = 20 if os.environ.get(“MJC_FAST”) == “1” else 100
def k_matrix(model, cam: int, width: int = W, height: int = H) -> np.ndarray: f = 0.5 * height / math.tan(math.radians(model.cam_fovy[cam]) / 2) return np.array([[f, 0, width / 2], [0, f, height / 2], [0, 0, 1]])
def back_project(model, data, cam: int, depth: np.ndarray, mask: np.ndarray) -> np.ndarray:
“"”World points (N, 3) of the pixels in mask, from metric depth along the optical axis.”””
k = k_matrix(model, cam, depth.shape[1], depth.shape[0])
v, u = np.nonzero(mask)
z = depth[v, u]
p_cv = np.c_[(u + 0.5 - k[0, 2]) * z / k[0, 0], (v + 0.5 - k[1, 2]) * z / k[1, 1], z]
r_wc = data.cam_xmat[cam].reshape(3, 3) @ FLIP
return p_cv @ r_wc.T + data.cam_xpos[cam]
def render(renderer, data, camera, kind: str) -> np.ndarray: renderer.disable_depth_rendering() renderer.disable_segmentation_rendering() if kind == “depth”: renderer.enable_depth_rendering() elif kind == “seg”: renderer.enable_segmentation_rendering() renderer.update_scene(data, camera) return renderer.render().copy()
def home(): model = mujoco.MjModel.from_xml_path(str(model_path(“pick_place”))) data = mujoco.MjData(model) mujoco.mj_resetDataKeyframe(model, data, model.key(“home”).id) mujoco.mj_forward(model, data) return model, data
def part1() -> None: model, data = home() renderer = mujoco.Renderer(model, H, W) cam = model.camera(“top”).id depth, seg = render(renderer, data, “top”, “depth”), render(renderer, data, “top”, “seg”) floor = (seg[…, 0] == model.geom(“floor”).id) & (seg[…, 1] == mujoco.mjtObj.mjOBJ_GEOM) k, height = k_matrix(model, cam), data.cam_xpos[cam][2] print(f” camera height {height:.3f} m; fovy {model.cam_fovy[cam]:.0f} deg”) print(f” {‘pixel (u, v)’:<16}{‘sees’:>7}{‘depth (m)’:>11}{‘ray length (m)’:>16}”) for u, v in ((W // 2, H // 2), (40, 40), (W - 40, 40), (40, H - 40), (W - 40, H - 40)): ray = height * math.hypot(1, (u + 0.5 - k[0, 2]) / k[0, 0], (v + 0.5 - k[1, 2]) / k[1, 1]) print(f” {str((u, v)):<16}{‘floor’ if floor[v, u] else ‘other’:>7}{depth[v, u]:>11.4f}{ray:>16.4f}”) renderer.close()
def plane_scene(height: float, znear: float) -> mujoco.MjModel: return mujoco.MjModel.from_xml_string(f”””
”””)
def part2() -> None: print(f” {‘height (m)’:>10}{‘max |error|, znear 0.01 m (um)’:>33}{‘znear 0.1 m (um)’:>19}{‘float32 spacing at z (um)’:>28}”) for height in (0.3, 1.0, 3.0, 10.0, 30.0): errors = [] for znear in (0.01, 0.1): model = plane_scene(height, znear) data = mujoco.MjData(model) mujoco.mj_forward(model, data) renderer = mujoco.Renderer(model, H, W) errors.append(1e6 * np.abs(render(renderer, data, “down”, “depth”) - height).max()) renderer.close() print(f” {height:>10.1f}{errors[0]:>33.2f}{errors[1]:>19.2f}{1e6 * np.spacing(np.float32(height)):>28.2f}”)
def part3() -> None: model, data = home() renderer = mujoco.Renderer(model, H, W) for label, hide_sites in ((“default scene options”, False), (“sites hidden”, True)): option = mujoco.MjvOption() if hide_sites: option.sitegroup[:] = 0 renderer.enable_segmentation_rendering() renderer.update_scene(data, “front”, scene_option=option) seg = renderer.render() renderer.disable_segmentation_rendering() types = {int(t): int(np.count_nonzero(seg[…, 1] == t)) for t in np.unique(seg[…, 1])} named = {mujoco.mjtObj(t).name if t >= 0 else “background”: n for t, n in types.items()} sites = {model.site(int(i)).name: int(np.count_nonzero((seg[…, 1] == mujoco.mjtObj.mjOBJ_SITE) & (seg[…, 0] == i))) for i in np.unique(seg[…, 0][seg[…, 1] == mujoco.mjtObj.mjOBJ_SITE])} print(f” {label:<24} pixels by type {named}; sites {sites}”) seg = render(renderer, data, “front”, “seg”) geoms = seg[…, 0][seg[…, 1] == mujoco.mjtObj.mjOBJ_GEOM] tray_geoms = {i for i in np.unique(geoms) if model.geom_bodyid[i] == model.body(“tray”).id} print(f” the tray is {len(tray_geoms)} visible geoms of one body; body-level mask: “ f”{int(np.isin(seg[…, 0], list(tray_geoms)).sum() if tray_geoms else 0)} pixels”) renderer.close()
def part4(width: int, height: int) -> None: model, data = home() renderer = mujoco.Renderer(model, height, width) cube, centre = model.geom(“red_cube”).id, data.body(“red_cube”).xpos.copy() clouds = [] print(f” {width} x {height}: red cube centre {np.round(centre, 3)} m, half-size 20 mm”) print(f” {‘camera’:<10}{‘floor points’:>13}{‘floor z max |err| (mm)’:>24}{‘cube points’:>13}{‘cube centroid minus centre (mm)’:>34}”) for name in (“front”, “side”, “top”): cam = model.camera(name).id depth, seg = render(renderer, data, name, “depth”), render(renderer, data, name, “seg”) is_geom = seg[…, 1] == mujoco.mjtObj.mjOBJ_GEOM floor = back_project(model, data, cam, depth, is_geom & (seg[…, 0] == model.geom(“floor”).id)) points = back_project(model, data, cam, depth, is_geom & (seg[…, 0] == cube)) clouds.append(points) print(f” {name:<10}{len(floor):>13d}{1000 * np.abs(floor[:, 2]).max():>24.3f}{len(points):>13d}” f”{np.array2string(1000 * (points.mean(axis=0) - centre), precision=1):>34}”) fused = np.vstack(clouds) print(f” {‘fused’:<10}{‘’:>13}{‘’:>24}{len(fused):>13d}{np.array2string(1000 * (fused.mean(axis=0) - centre), precision=1):>34}”) box = (fused.min(axis=0) + fused.max(axis=0)) / 2 print(f” {‘fused, centre of the bounding box’:<60}{np.array2string(1000 * (box - centre), precision=1):>34}”) renderer.close()
def part5(arm_aside: bool) -> None: model, data = home() renderer = mujoco.Renderer(model, H, W) cam, cube = model.camera(“top”).id, model.geom(“red_cube”).id adr = model.jnt_qposadr[model.joint(“red_cube”).id] rng = np.random.default_rng(10) xy = np.c_[rng.uniform(0.40, 0.60, STATES), rng.uniform(-0.20, 0.00, STATES)] yaw = rng.uniform(-math.pi / 4, math.pi / 4, STATES) errors, hidden = [], 0 for (x, y), a in zip(xy, yaw): mujoco.mj_resetDataKeyframe(model, data, model.key(“home”).id) if arm_aside: data.qpos[0] = 1.6 # base joint: swing the arm to the robot’s left for name, other in ((“green_cube”, (-0.6, 0.6)), (“blue_cube”, (-0.6, -0.6))): # set aside, as in 10.2 o = model.jnt_qposadr[model.joint(name).id] data.qpos[o:o + 3] = [*other, 0.02] data.qpos[adr:adr + 7] = [x, y, 0.02, math.cos(a / 2), 0, 0, math.sin(a / 2)] mujoco.mj_forward(model, data) depth, seg = render(renderer, data, “top”, “depth”), render(renderer, data, “top”, “seg”) points = back_project(model, data, cam, depth, (seg[…, 1] == mujoco.mjtObj.mjOBJ_GEOM) & (seg[…, 0] == cube)) top_face = points[points[:, 2] > 0.039] # the top face is at z = 0.04 if len(top_face) < 0.8 * 0.04**2 / (1.36 / k_matrix(model, cam)[0, 0]) ** 2: hidden += 1 # less than 80% of the face visible continue errors.append(1000 * np.linalg.norm(top_face[:, :2].mean(axis=0) - [x, y])) errors = np.array(errors) print(f” arm {‘swung aside (joint 1 at 1.6 rad)’ if arm_aside else ‘at home’:<34} top face less than 80% visible: {hidden:3d}/{STATES};” f” xy error (mm): mean {errors.mean():.2f}, 95th percentile {np.percentile(errors, 95):.2f}, max {errors.max():.2f}”) renderer.close()
if name == “main”: print(f”MuJoCo {mujoco.version}; renders {W} x {H}”) print(“(1) depth at floor pixels of the top camera, home state”) part1() print(“(2) depth error against distance (plane, camera looking straight down, extent 1 m)”) part2() print(“(3) segmentation of the front camera”) part3() print(“(4) point clouds from depth, home state”) part4(W, H) part4(320, 240) print(f”(5) the top camera as a cube-position sensor: {STATES} cube poses (seed 10, Lesson 10.2’s distribution),”) print(f” xy centre of the top face’s points; pixel footprint on the face {1000 * 1.36 * 2 * math.tan(math.radians(22.5)) / H:.2f} mm”) part5(arm_aside=False) part5(arm_aside=True)
Output:
```text
MuJoCo 3.14.0; renders 640 x 480
(1) depth at floor pixels of the top camera, home state
camera height 1.400 m; fovy 45 deg
pixel (u, v) sees depth (m) ray length (m)
(320, 240) other 0.8086 1.4000
(40, 40) floor 1.4000 1.6274
(600, 40) floor 1.4000 1.6284
(40, 440) floor 1.4000 1.6281
(600, 440) floor 1.4000 1.6291
(2) depth error against distance (plane, camera looking straight down, extent 1 m)
height (m) max |error|, znear 0.01 m (um) znear 0.1 m (um) float32 spacing at z (um)
0.3 0.00 0.00 0.03
1.0 0.00 0.00 0.12
3.0 0.24 0.24 0.24
10.0 0.00 0.95 0.95
30.0 1.91 1.91 1.91
(3) segmentation of the front camera
default scene options pixels by type {'mjOBJ_GEOM': 302532, 'mjOBJ_SITE': 4668}; sites {'left_zone': 1965, 'right_zone': 2599, 'tray_center': 104}
sites hidden pixels by type {'mjOBJ_GEOM': 307200}; sites {}
the tray is 5 visible geoms of one body; body-level mask: 8281 pixels
(4) point clouds from depth, home state
640 x 480: red cube centre [0.45 0.12 0.02] m, half-size 20 mm
camera floor points floor z max |err| (mm) cube points cube centroid minus centre (mm)
front 282194 0.000 601 [11. -1.2 8.2]
side 242369 0.000 466 [ -0.2 -13.3 6.9]
top 277331 0.000 339 [ 0.5 -1.3 16.6]
fused 1406 [ 4.8 -5.2 9.8]
fused, centre of the bounding box [ 0. -0. 0.2]
320 x 240: red cube centre [0.45 0.12 0.02] m, half-size 20 mm
camera floor points floor z max |err| (mm) cube points cube centroid minus centre (mm)
front 70589 0.000 148 [11.4 -0.9 8.5]
side 60492 0.000 108 [ 0.6 -14.5 6.3]
top 69372 0.000 81 [ 0.7 -0.3 20. ]
fused 337 [ 5.4 -5.1 10.5]
fused, centre of the bounding box [ 0.4 -0.1 1. ]
(5) the top camera as a cube-position sensor: 100 cube poses (seed 10, Lesson 10.2's distribution),
xy centre of the top face's points; pixel footprint on the face 2.35 mm
arm at home top face less than 80% visible: 34/100; xy error (mm): mean 0.23, 95th percentile 0.67, max 2.28
arm swung aside (joint 1 at 1.6 rad) top face less than 80% visible: 0/100; xy error (mm): mean 0.15, 95th percentile 0.55, max 0.99
Point clouds bulge outward toward the image corners. Depth treated as ray length, or ray length treated as depth. MuJoCo returns depth along the optical axis.
A wall of points at 80 m. Pixels that hit nothing report the far clipping plane (Lesson 3.2). Mask them out using segmentation (id −1) before back-projecting.
The floor mask has holes where nothing is in front of it. Sites are segmented. Hide the site groups in MjvOption, or treat site pixels as background.
An object’s estimated position is off by half its size, toward the camera. The centroid of its visible points. Estimate from a face you know you see, fit the shape, or fuse views.
Object masks are split into pieces. The object is several geoms. Build body-level masks through model.geom_bodyid.
Repeat output (5) with the top camera at 320 × 240 and at 160 × 120. Predict how the error scales with the pixel footprint (2.35 mm at 640 × 480), then measure it. At what resolution does the 95th-percentile error exceed 2 mm?
Write a depth-noise model for a structured-light sensor and measure its effect on output (5): Gaussian noise with a standard deviation growing as $z^2$, pixels dropped where the surface is steeply inclined to the ray, and the depth quantized to 1 mm. State your model’s parameters, the source you took them from, and how the cube-position error changes.
[!research] Simulated depth is too good Rendered depth in this test is exact to float32 rounding, while a real depth camera’s error grows with distance and fails at edges and on dark or shiny surfaces. A policy or estimator trained on perfect simulated depth meets a different input distribution on hardware, and edge pixels, where real sensors fail most, are exactly where grasp-relevant geometry lives (inference). Two families of remedies exist: adding sensor-specific noise and dropout models to simulated depth, and learning on representations less sensitive to those errors, such as segmented point clouds or features from pretrained vision models. Level 18 builds the first; Level 15 compares observation choices.
{"id": "11.2-check", "title": "Knowledge check", "questions": [
{"kind": "numeric", "q": "A pixel has normalized coordinates (a, b) = (0.3, 0.4), and MuJoCo reports depth 2.0 m. How far is the point from the camera (m)?",
"answer": 2.236, "tol": 0.002, "unit": "m",
"explain": "<p>$z\\sqrt{1 + a^2 + b^2} = 2\\sqrt{1.25}$ = 2.236 m. The 2.0 m is along the optical axis.</p>"},
{"kind": "numeric", "q": "A 40 mm cube seen only from directly above: by how many millimetres is the centroid of its points above its centre?",
"answer": 20, "tol": 0.5, "unit": "mm",
"explain": "<p>Only the top face is visible, at half the height above the centre (output 4, top camera at 320 × 240: 20.0 mm).</p>"},
{"kind": "mcq", "q": "Pixels on the floor of a segmentation image are labelled with an object type other than geom. What are they most likely?",
"options": ["Rendering errors", "Sites that are drawn (for example translucent region markers)", "Pixels beyond the far plane", "Joints"],
"answer": 1,
"explain": "<p>Output (3): drawn sites are segmented; hide their groups through the scene options.</p>"},
{"kind": "mcq", "q": "Three cameras see a cube. Which estimate of its centre did output (4) find closest to the truth?",
"options": ["The centroid of the top camera's points", "The centroid of all fused points", "The centre of the fused points' bounding box", "The centroid of the front camera's points"],
"answer": 2,
"explain": "<p>0.2 mm for the bounding box against 4.8 to 9.8 mm per axis for the fused centroid.</p>"}
]}
Lesson 11.3 records all of this, with states and actions, into a dataset whose every frame can be checked: a schema, reproducible generation, and a validator.