A MuJoCo camera is a pinhole: a point in the world reaches the image through a rigid transform into the camera’s frame (the extrinsics) and a perspective division scaled by the focal length and shifted by the principal point (the intrinsics). Every use of images in this course depends on getting both right: datasets record them, point clouds invert them, a policy trained on simulated images assumes them, and sim-to-real transfer requires the simulated camera to match a calibrated real one.
The model is textbook; the conventions are where errors live. MuJoCo’s camera frame differs from OpenCV’s, its field of view is vertical, its principal-point offset has its own sign, and the render size changes the intrinsics in different ways depending on how the camera was defined. This lesson writes the model down, then checks it against renders to a fraction of a pixel, and measures what each convention mistake costs.
[!established] MuJoCo cameras A camera looks along its own −z axis with +y up (MuJoCo convention, documented in the XML reference). Its pose in the world is
data.cam_xposanddata.cam_xmat(the columns of the 3 × 3 matrix are the camera’s axes in world coordinates).fovyis the vertical field of view in degrees. A camera can instead be given a sensor:resolution,sensorsize, andfocalorfocalpixel, optionallyprincipalorprincipalpixel, in which case MuJoCo derivesfovyfrom them; the compiled values are inmodel.cam_intrinsicandmodel.cam_sensorsize.
The lab renders the selected model camera of the pick-and-place scene and overlays, for each cube, its centre projected with the pinhole model (cross) and the centroid of its segmentation mask (white circle).
fovy camera keeps its vertical field of view and square pixels; the horizontal field shrinks from 57.8° to 45°.```lab camera {“dock”: true, “title”: “A camera, its image and the pinhole model”, “height”: 260}
## Mathematics
### From the world to a pixel
Write the camera's pose as $R_{wc} = $ `cam_xmat` and $t_{wc} = $ `cam_xpos`. In OpenCV's convention (x right, y down, z forward), which most vision code expects, the camera axes are MuJoCo's with y and z negated, $R_{wc}^{cv} = R_{wc}\,\mathrm{diag}(1,-1,-1)$. A world point $p$ maps to
$$p_c = R\,p + t,\qquad R = (R_{wc}^{cv})^\top,\quad t = -R\,t_{wc},$$
$$u = f_x\frac{x_c}{z_c} + c_x,\qquad v = f_y\frac{y_c}{z_c} + c_y,$$
with $u$ to the right and $v$ down, and pixel $(i, j)$ covering $[i, i+1) \times [j, j+1)$, so its centre is at $(i + 0.5, j + 0.5)$. For a camera defined by `fovy`, rendered $W \times H$:
$$f_x = f_y = \frac{H/2}{\tan(\text{fovy}/2)},\qquad c_x = W/2,\quad c_y = H/2.$$
At 640 × 480 and fovy = 45°, $f$ = 579.4 px. Output (1, 2) projects 16 marker spheres (radius 5 mm) with this model and compares them with the centroids of their masks rendered by `mujoco.Renderer`: mean errors of 0.08 to 0.14 px and at most 0.44 px over the four cameras, which is the precision of a mask centroid, not an error of the model.
### What the conventions cost
Output (3) repeats the comparison under five common mistakes:
| Mistake | Mean error | Max error |
|---|---|---|
| none | 0.12 px | 0.21 px |
| pixel centres at $i$ instead of $i + 0.5$ | 0.72 px | 0.91 px |
| fovy taken as the horizontal field of view | 28.8 px | 43.2 px |
| the hand body's position used for the wrist camera | 61.7 px | 90.1 px |
| MuJoCo camera axes used as OpenCV axes | 136 px | 222 px |
| `cam_xmat` used without transposing | 1524 px | 2803 px |
The half-pixel convention is the only mistake small enough to survive a visual check, and it is enough to fail a 0.5 px calibration tolerance. The wrist camera is offset from the hand's origin by 5.5 cm along the hand's x axis and 2 cm along its z axis, so using the body's position puts every point 60 to 90 px away. The last two are gross, but they produce plausible-looking images when the projected points happen to land inside the frame.
### The render size changes the intrinsics
The intrinsics belong to the camera and the image size together. Output (4) fits $f_x, f_y, c_x, c_y$ from renders at two sizes:
- a `fovy` camera keeps $f_x = f_y$ = 579.4 px at 640 × 480 and at 480 × 480, so its horizontal field of view changes, 57.8° to 45.0°;
- a camera defined by a 4:3 sensor and a 600 px focal length keeps its field of view (56.1° by 43.6°) at both sizes, so rendered at 480 × 480 its pixels are no longer square: $f_x$ = 450 px, $f_y$ = 600 px.
Neither is wrong; they are different contracts. A dataset that records only "fovy" or only "the camera's K" loses this distinction. Record the K of the rendered image.
### A calibrated camera in MuJoCo
A real camera calibrated with OpenCV gives $f_x, f_y, c_x, c_y$ at its resolution $W \times H$. MuJoCo accepts the same numbers with one convention to respect: rendered at its own resolution, a camera with `principalpixel` $(p_x, p_y)$ has its principal point at
$$c_x = W/2 - p_x,\qquad c_y = H/2 - p_y,$$
measured in output (5): an offset of 30 px along x alone moved the fitted $c_x$ from 320 to 290, and 30 px along y alone moved $c_y$ from 240 to 210. So the attributes are `resolution` = $(W, H)$, `focalpixel` = $(f_x, f_y)$, `principalpixel` = $(W/2 - c_x,\ H/2 - c_y)$, and `sensorsize` = $(W, H)$ times any pixel pitch. Output (5) does this for $K$ with $f_x$ = 615.3, $f_y$ = 614.8, $c_x$ = 326.4, $c_y$ = 238.1 and fits back 615.39, 614.78, 326.39, 238.15 from the render (reprojection 0.12 px mean); with the opposite sign the principal point lands mirrored about the centre (313.6, 241.9) and every point is 13.4 px off.
> [!recommendation] Check a camera before you trust it
> Put a few small spheres at known positions, render segmentation, and compare mask centroids with your projection. Sub-pixel agreement means your intrinsics, extrinsics and conventions are right; anything else points to the mistake by its size, as in the table above. It takes a dozen lines and catches every error on this page.
## Implementation
```io
INPUT: pick_place.xml with 16 marker spheres; an OpenCV-style K
PROCESS: project markers with the pinhole model and compare with mask centroids per camera, under five mistakes, at two render sizes, and for cameras built from K
OUTPUT: printed tables
```python file=examples/l11_1_camera_model.py “"”Lesson 11.1: the camera model: intrinsics, extrinsics, projection, checked against renders.
INPUT pick_place.xml with 16 marker spheres (radius 5 mm) added by MjSpec at known positions PROCESS (1) intrinsics of the scene’s four cameras from fovy and the render size, extrinsics from cam_xpos and cam_xmat in the OpenCV convention (x right, y down, z forward); (2) project every marker and compare with the centroid of its segmentation mask rendered at 640 x 480: reprojection error per camera; (3) the same comparison under five common mistakes; (4) what the render’s aspect ratio does to a fovy camera and to a camera defined by a sensor size: intrinsics fitted from renders at 640 x 480 and 480 x 480; (5) an OpenCV calibration matrix K turned into MuJoCo camera attributes, rendered, and K fitted back from the render, with the right and the wrong principal-point sign; (6) the lab’s readout: each cube’s projected centre against its mask centroid at 320 x 240 OUTPUT printed tables
Run: MUJOCO_GL=egl python examples/l11_1_camera_model.py (or osmesa without a GPU) “””
import math
import mujoco import numpy as np
from mjcourse import model_path
W, H = 640, 480 FLIP = np.diag([1.0, -1.0, -1.0]) # MuJoCo camera axes (x right, y up, looking along -z) -> OpenCV MARKER = 0.005 # marker radius (m)
def scene_with_markers(seed: int = 0, extra_cameras=()) -> tuple[mujoco.MjModel, np.ndarray]: “"”pick_place plus 16 non-colliding spheres of radius 5 mm over the workspace.””” spec = mujoco.MjSpec.from_file(str(model_path(“pick_place”))) rng = np.random.default_rng(seed) xs, ys = np.meshgrid(np.linspace(0.35, 0.65, 4), np.linspace(-0.18, 0.18, 4)) points = np.c_[xs.ravel(), ys.ravel(), np.zeros(16)] + np.c_[rng.uniform(-0.02, 0.02, (16, 2)), rng.uniform(0.03, 0.2, 16)] for i, p in enumerate(points): spec.worldbody.add_geom(name=f”marker{i}”, type=mujoco.mjtGeom.mjGEOM_SPHERE, size=[MARKER, 0, 0], pos=p, contype=0, conaffinity=0, rgba=[1, 1, 0, 1]) for add in extra_cameras: add(spec) return spec.compile(), points
def intrinsics_fovy(model: mujoco.MjModel, cam: int, width: int, height: int) -> np.ndarray: “"”K for a camera defined by fovy: square pixels, principal point at the image centre.””” f = 0.5 * height / math.tan(math.radians(model.cam_fovy[cam]) / 2) return np.array([[f, 0, width / 2], [0, f, height / 2], [0, 0, 1]])
def extrinsics(data: mujoco.MjData, cam: int) -> tuple[np.ndarray, np.ndarray]: “"”World-to-camera rotation and translation in the OpenCV convention: p_cam = R p_world + t.””” r_wc = data.cam_xmat[cam].reshape(3, 3) @ FLIP # columns: OpenCV camera axes in the world return r_wc.T, -r_wc.T @ data.cam_xpos[cam]
def project(k: np.ndarray, r: np.ndarray, t: np.ndarray, points: np.ndarray) -> np.ndarray: p = points @ r.T + t uv = p @ k.T return uv[:, :2] / uv[:, 2:3]
def masks(model, data, camera: str, n_markers: int, width: int = W, height: int = H, half: float = 0.5): “"”Centroid (u, v) of each marker’s mask with pixel centres at index + half (NaN if not visible or touching the border), and the mask’s pixel count.””” renderer = mujoco.Renderer(model, height, width) renderer.enable_segmentation_rendering() renderer.update_scene(data, camera) seg = renderer.render() renderer.close() uv, counts = np.full((n_markers, 2), np.nan), np.zeros(n_markers) for i in range(n_markers): ys, xs = np.nonzero((seg[…, 0] == model.geom(f”marker{i}”).id) & (seg[…, 1] == mujoco.mjtObj.mjOBJ_GEOM)) counts[i] = len(xs) if len(xs) >= 6 and xs.min() > 0 and ys.min() > 0 and xs.max() < width - 1 and ys.max() < height - 1: uv[i] = xs.mean() + half, ys.mean() + half return uv, counts
def unoccluded(model, data, cam: int, points, uv, counts, height: int = H) -> np.ndarray: “"”Markers whose mask has at least 80% of the pixels an unoccluded sphere would cover.””” r, t = extrinsics(data, cam) z = (points @ r.T + t)[:, 2] f = 0.5 * height / math.tan(math.radians(model.cam_fovy[cam]) / 2) return ~np.isnan(uv[:, 0]) & (counts >= 0.8 * math.pi * (f * MARKER / z) ** 2)
def errors(uv_model, uv_mask, ok) -> tuple[float, float]: e = np.linalg.norm(uv_model[ok] - uv_mask[ok], axis=1) return float(e.mean()), float(e.max())
def fit_k(points, r, t, uv) -> np.ndarray: “"”Least-squares fx, fy, cx, cy from known extrinsics and measured pixels.””” p = points @ r.T + t a, b = p[:, 0] / p[:, 2], p[:, 1] / p[:, 2] ok = ~np.isnan(uv[:, 0]) fx, cx = np.linalg.lstsq(np.c_[a, np.ones_like(a)][ok], uv[ok, 0], rcond=None)[0] fy, cy = np.linalg.lstsq(np.c_[b, np.ones_like(b)][ok], uv[ok, 1], rcond=None)[0] return np.array([fx, fy, cx, cy])
def part1_2(model, data, points) -> None: print(f” {‘camera’:<14}{‘fovy (deg)’:>11}{‘f (px)’:>9}{‘fov_x (deg)’:>12}{‘position (m)’:>24}{‘markers’:>9}{‘mean err (px)’:>15}{‘max err (px)’:>14}”) for cam in range(model.ncam): name = model.camera(cam).name k = intrinsics_fovy(model, cam, W, H) r, t = extrinsics(data, cam) uv_mask, counts = masks(model, data, name, len(points)) ok = unoccluded(model, data, cam, points, uv_mask, counts) fov_x = 2 * math.degrees(math.atan(W / 2 / k[0, 0])) mean, mx = errors(project(k, r, t, points), uv_mask, ok) if ok.any() else (float(“nan”), float(“nan”)) print(f” {name:<14}{model.cam_fovy[cam]:>11.1f}{k[0, 0]:>9.1f}{fov_x:>12.1f}{np.array2string(data.cam_xpos[cam], precision=3):>24}” f”{ok.sum():>6d}/16{mean:>15.3f}{mx:>14.3f}”)
def part3(model, data, points) -> None: cam = model.camera(“front”).id k, (r, t) = intrinsics_fovy(model, cam, W, H), extrinsics(data, cam) uv_mask, counts = masks(model, data, “front”, len(points)) ok = unoccluded(model, data, cam, points, uv_mask, counts) k_hfov = k.copy() k_hfov[0, 0] = k_hfov[1, 1] = 0.5 * W / math.tan(math.radians(model.cam_fovy[cam]) / 2) # fovy read as horizontal r_mj = data.cam_xmat[cam].reshape(3, 3).T # MuJoCo axes, no flip cases = [(“correct”, project(k, r, t, points), uv_mask), (“fovy taken as the horizontal field of view”, project(k_hfov, r, t, points), uv_mask), (“MuJoCo camera axes used as OpenCV axes”, project(k, r_mj, -r_mj @ data.cam_xpos[cam], points), uv_mask), (“mask centroid without the half-pixel offset”, project(k, r, t, points), masks(model, data, “front”, len(points), half=0.0)[0]), (“cam_xmat used without transposing”, project(k, r.T, -r.T @ data.cam_xpos[cam], points), uv_mask)] for label, uv, measured in cases: e = np.linalg.norm(uv[ok] - measured[ok], axis=1) e = np.where(np.isfinite(e), e, np.inf) print(f” {label:<46} mean {np.mean(e):9.2f} px max {np.max(e):9.2f} px”) wrist = model.camera(“gripper/wrist”).id k_w, (r_w, t_w) = intrinsics_fovy(model, wrist, W, H), extrinsics(data, wrist) hand = model.cam_bodyid[wrist] uv_w, counts_w = masks(model, data, “gripper/wrist”, len(points)) ok_w = unoccluded(model, data, wrist, points, uv_w, counts_w) for label, (rr, tt) in ((“wrist camera, its own pose”, (r_w, t_w)), (“wrist camera, the hand body’s position instead”, (r_w, -r_w @ data.xpos[hand]))): e = np.linalg.norm(project(k_w, rr, tt, points)[ok_w] - uv_w[ok_w], axis=1) print(f” {label:<46} mean {np.mean(e):9.2f} px max {np.max(e):9.2f} px ({ok_w.sum()} markers)”)
def sensor_camera(name: str, k: np.ndarray | None, sign: float = 1.0, pitch: float = 3e-6, principal=None): “"”A camera at the front camera’s pose defined by focal and principal point in pixels. MuJoCo’s principal-point offset (px, py) puts the image centre at (W/2 - px, H/2 - py).””” base = mujoco.MjModel.from_xml_path(str(model_path(“pick_place”))) front = base.camera(“front”).id # compiled pose: the XML gives it as xyaxes
def add(spec: mujoco.MjSpec) -> None:
cam = spec.worldbody.add_camera(name=name, pos=base.cam_pos[front], quat=base.cam_quat[front])
fx, fy, cx, cy = (600.0, 600.0, W / 2, H / 2) if k is None else (k[0, 0], k[1, 1], k[0, 2], k[1, 2])
cam.resolution = [W, H]
cam.sensor_size = [W * pitch, H * pitch]
cam.focal_pixel = [fx, fy]
cam.principal_pixel = list(principal) if principal is not None else [sign * (W / 2 - cx), sign * (H / 2 - cy)]
return add
def part4_5() -> None: k_cal = np.array([[615.3, 0, 326.4], [0, 614.8, 238.1], [0, 0, 1]]) model, points = scene_with_markers(extra_cameras=(sensor_camera(“sensor”, None), sensor_camera(“calibrated”, k_cal), sensor_camera(“calibrated_wrong_sign”, k_cal, sign=-1.0), sensor_camera(“offset_x”, None, principal=(30, 0)), sensor_camera(“offset_y”, None, principal=(0, 30)))) data = mujoco.MjData(model) mujoco.mj_resetDataKeyframe(model, data, model.key(“home”).id) mujoco.mj_forward(model, data) print(“(4) intrinsics fitted from renders: fx, fy, cx, cy (px)”) for name in (“front”, “sensor”): cam = model.camera(name).id r, t = extrinsics(data, cam) for width, height in ((640, 480), (480, 480)): fitted = fit_k(points, r, t, masks(model, data, name, len(points), width, height)[0]) fov_x = 2 * math.degrees(math.atan(width / 2 / fitted[0])) fov_y = 2 * math.degrees(math.atan(height / 2 / fitted[1])) kind = “fovy 45” if name == “front” else “sensor 4:3, focal 600 px” print(f” {kind:<26} render {width} x {height}: {np.array2string(fitted, precision=2):<34} fov_x {fov_x:5.1f}, fov_y {fov_y:5.1f} deg”) print(“(5) the principal-point offset’s sign: focal 600 px, principal_pixel set, cx and cy fitted from renders”) for name, offset in ((“sensor”, (0, 0)), (“offset_x”, (30, 0)), (“offset_y”, (0, 30))): cam = model.camera(name).id r, t = extrinsics(data, cam) fitted = fit_k(points, r, t, masks(model, data, name, len(points))[0]) print(f” principal_pixel {str(offset):<9} fitted cx {fitted[2]:7.2f}, cy {fitted[3]:7.2f}”) print(“ an OpenCV K into MuJoCo: fx 615.3, fy 614.8, cx 326.4, cy 238.1 at 640 x 480”) for name, label in ((“calibrated”, “principal_pixel = (W/2 - cx, H/2 - cy)”), (“calibrated_wrong_sign”, “principal_pixel = (cx - W/2, cy - H/2)”)): cam = model.camera(name).id r, t = extrinsics(data, cam) uv = masks(model, data, name, len(points))[0] fitted = fit_k(points, r, t, uv) ok = ~np.isnan(uv[:, 0]) e = np.linalg.norm(project(k_cal, r, t, points)[ok] - uv[ok], axis=1) print(f” {label:<40} fitted {np.array2string(fitted, precision=2):<34} reprojection with K: mean {e.mean():.2f}, max {e.max():.2f} px”)
def part6() -> None: model = mujoco.MjModel.from_xml_path(str(model_path(“pick_place”))) data = mujoco.MjData(model) mujoco.mj_resetDataKeyframe(model, data, model.key(“home”).id) mujoco.mj_forward(model, data) width, height = 320, 240 renderer = mujoco.Renderer(model, height, width) renderer.enable_segmentation_rendering() for cam in range(model.ncam): renderer.update_scene(data, cam) seg = renderer.render() k, (r, t) = intrinsics_fovy(model, cam, width, height), extrinsics(data, cam) row = [] for name in (“red_cube”, “green_cube”, “blue_cube”): ys, xs = np.nonzero((seg[…, 0] == model.geom(name).id) & (seg[…, 1] == mujoco.mjtObj.mjOBJ_GEOM)) if len(xs) == 0: row.append(f”{name.split(‘’)[0]} not visible”) continue uv = project(k, r, t, data.body(name).xpos[None])[0] row.append(f”{name.split(‘’)[0]} {np.hypot(uv[0] - xs.mean() - 0.5, uv[1] - ys.mean() - 0.5):.2f} ({len(xs)} px)”) print(f” {model.camera(cam).name:<14}” + “; “.join(row)) renderer.close()
if name == “main”: model, points = scene_with_markers() data = mujoco.MjData(model) mujoco.mj_resetDataKeyframe(model, data, model.key(“home”).id) mujoco.mj_forward(model, data) print(f”MuJoCo {mujoco.version}; renders {W} x {H}; 16 markers of radius {1000 * MARKER:.0f} mm”) print(“(1, 2) model cameras: intrinsics from fovy, extrinsics from cam_xpos/cam_xmat, reprojection against masks”) part1_2(model, data, points) print(“(3) common mistakes, front camera (wrist camera for the last two rows)”) part3(model, data, points) part4_5() print(“(6) cube centres projected against their mask centroids, 320 x 240, home state (px)”) part6()
Output:
```text
MuJoCo 3.14.0; renders 640 x 480; 16 markers of radius 5 mm
(1, 2) model cameras: intrinsics from fovy, extrinsics from cam_xpos/cam_xmat, reprojection against masks
camera fovy (deg) f (px) fov_x (deg) position (m) markers mean err (px) max err (px)
front 45.0 579.4 57.8 [1.35 0. 0.75] 16/16 0.116 0.214
side 45.0 579.4 57.8 [ 0.5 -1. 0.6] 16/16 0.124 0.228
top 45.0 579.4 57.8 [0.5 0. 1.4] 11/16 0.083 0.157
gripper/wrist 75.0 312.8 91.3 [0.501 0. 0.384] 15/16 0.141 0.440
(3) common mistakes, front camera (wrist camera for the last two rows)
correct mean 0.12 px max 0.21 px
fovy taken as the horizontal field of view mean 28.77 px max 43.23 px
MuJoCo camera axes used as OpenCV axes mean 135.71 px max 221.80 px
mask centroid without the half-pixel offset mean 0.72 px max 0.91 px
cam_xmat used without transposing mean 1524.36 px max 2802.57 px
wrist camera, its own pose mean 0.14 px max 0.44 px (15 markers)
wrist camera, the hand body's position instead mean 61.68 px max 90.06 px (15 markers)
(4) intrinsics fitted from renders: fx, fy, cx, cy (px)
fovy 45 render 640 x 480: [579.36 578.99 320. 239.98] fov_x 57.8, fov_y 45.0 deg
fovy 45 render 480 x 480: [579.36 578.99 240. 239.98] fov_x 45.0, fov_y 45.0 deg
sensor 4:3, focal 600 px render 640 x 480: [600.16 600.14 320. 240.02] fov_x 56.1, fov_y 43.6 deg
sensor 4:3, focal 600 px render 480 x 480: [449.96 599.62 240.05 240.02] fov_x 56.1, fov_y 43.6 deg
(5) the principal-point offset's sign: focal 600 px, principal_pixel set, cx and cy fitted from renders
principal_pixel (0, 0) fitted cx 320.00, cy 240.02
principal_pixel (30, 0) fitted cx 290.00, cy 240.02
principal_pixel (0, 30) fitted cx 320.00, cy 210.02
an OpenCV K into MuJoCo: fx 615.3, fy 614.8, cx 326.4, cy 238.1 at 640 x 480
principal_pixel = (W/2 - cx, H/2 - cy) fitted [615.39 614.78 326.39 238.15] reprojection with K: mean 0.12, max 0.26 px
principal_pixel = (cx - W/2, cy - H/2) fitted [615.18 614.56 313.61 241.91] reprojection with K: mean 13.35, max 13.57 px
(6) cube centres projected against their mask centroids, 320 x 240, home state (px)
front red 0.22 (148 px); green 0.24 (165 px); blue 0.21 (150 px)
side red 0.25 (108 px); green 0.26 (147 px); blue 0.36 (189 px)
top red 0.31 (81 px); green 5.31 (2 px); blue 0.25 (80 px)
gripper/wrist red 1.12 (429 px); green 0.77 (376 px); blue 0.88 (480 px)
Part (6) is what the lab shows: at 320 × 240 the cubes’ projected centres lie within 0.2 to 0.4 px of their mask centroids for the three fixed cameras. On the top camera the green cube is almost entirely hidden by the arm (2 pixels visible), and on the wrist camera the cubes are close enough that their visible faces’ centroid is about a pixel from the projected centre.
Projected points are mirrored or missing. The MuJoCo-to-OpenCV axis flip is missing (or applied twice): points come out with negative depth. Check that $z_c > 0$ for something you can see.
Everything is off by a scale factor about the centre. The focal length is wrong: fovy taken as horizontal, the wrong image height, or intrinsics computed for one render size and used at another.
A constant offset of half a pixel. Pixel-centre convention. Decide whether pixel $i$ is centred at $i$ or $i + 0.5$, and use the same convention in projection, in mask centroids and in back-projection.
A wrist camera is off by tens of pixels and the error moves with the arm. The camera’s pose taken from its body (xpos, xmat) instead of cam_xpos, cam_xmat.
A calibrated camera’s image is shifted. The principal-point sign: MuJoCo’s offset $(p_x, p_y)$ puts the centre at $(W/2 - p_x, H/2 - p_y)$.
Render the front camera at 320 × 240 and 1280 × 960 (raise offwidth and offheight in the model’s <visual><global> first) and compute the focal length for each from fovy. Predict how the projected marker positions scale, then check with the script’s masks and project functions.
Calibrate a MuJoCo camera the way you would a real one. Render a checkerboard (a textured box, or a grid of small spheres) from 15 random poses of the board, detect the corners or sphere centroids, and run cv2.calibrateCamera. Compare the recovered $K$ with the one the camera’s attributes imply, and report how the error depends on the number of views and the board’s coverage of the image.
[!research] Intrinsics as a domain gap Policies trained on simulated images are often deployed on cameras whose intrinsics differ from the simulated ones by a few percent in focal length or tens of pixels in principal point. Since a fixed pixel offset corresponds to a different 3-D offset at each depth, such a mismatch is not removed by image augmentation that shifts or crops uniformly (inference). Two remedies follow from this lesson: build the simulated camera from the real calibration, as in output (5), or randomize $f_x, f_y, c_x, c_y$ during training over the range of the real cameras (Level 18). The first is cheap when the deployment camera is known; the second when it is not.
{"id": "11.1-check", "title": "Knowledge check", "questions": [
{"kind": "numeric", "q": "A camera has fovy = 60° and is rendered at 640 × 480. What is its focal length in pixels?",
"answer": 415.7, "tol": 0.3, "unit": "px",
"explain": "<p>$f = (H/2)/\\tan(30°) = 240/0.5774$ = 415.7 px. The width does not enter.</p>"},
{"kind": "numeric", "q": "The same camera rendered at 480 × 480: what is its horizontal field of view in degrees?",
"answer": 60.0, "tol": 0.1, "unit": "deg",
"explain": "<p>Square pixels and the same $f$: $2\\arctan(240/415.7)$ = 60°. At 640 × 480 it would be 75.2°.</p>"},
{"kind": "numeric", "q": "A calibration gives cx = 330.5 for a 640-pixel-wide image. What value of principalpixel x makes MuJoCo render with that principal point?",
"answer": -10.5, "tol": 0.01, "unit": "px",
"explain": "<p>$p_x = W/2 - c_x = 320 - 330.5$ = −10.5.</p>"},
{"kind": "mcq", "q": "Your projected points land 0.5 px from mask centroids in both u and v, consistently. Which mistake is this?",
"options": ["Wrong focal length", "Pixel-centre convention", "Missing axis flip", "Wrong camera pose"],
"answer": 1,
"explain": "<p>A constant half-pixel offset in both axes is the pixel-centre convention (output 3: 0.72 px mean).</p>"}
]}
Lesson 11.2 turns depth and segmentation into geometry: what MuJoCo’s depth measures, point clouds in the world frame, and what a single view can and cannot tell you about an object’s position.