A pick-and-place is a sequence of motions joined by decisions. Scripted, it is a state machine over 6-D task-space targets: approach above the object, descend, close, verify the grasp, lift, verify again, carry, lower, release, retreat, and finally judge where the object ended up. Each target becomes joint angles by inverse kinematics (Lesson 6.3), and a joint controller with gravity compensation (Lesson 8.2) follows a minimum-jerk path to them.
Two habits separate a pipeline you can trust from a demonstration that worked once. Every transition checks the world, not the plan: the arm reaching the grasp pose says nothing about whether the cube is between the pads, while the contacts and the gripper opening do. And success is a rate measured over randomized initial states, reported with a confidence interval and broken down by failure category, because a single video samples one point of a distribution.
[!established] The task and its success criterion in this lesson Scene
pick_place.xml: the 7-DOF arm with the parallel gripper, a 40 mm red cube placed uniformly in a 20 × 20 cm area (x 0.40 to 0.60 m, y −0.20 to 0 m) with yaw uniform in ±45°, and a tray centred at (0.62, 0.20) m. An episode succeeds when, after the gripper has let go and retreated, the cube’s centre lies within 7 cm of the tray centre in x and y and below 6 cm. Rates are per episode over 100 initial states drawn with seed 10; intervals are 95% Wilson intervals.
Run an episode places the red cube at a new random pose and runs the full pipeline in the page. The perception error slider adds Gaussian noise of the chosen standard deviation to each axis of the cube’s estimated position, the way a camera-based pose estimate would be off. The 4x speed button shortens the wait.
```lab pickplace {“dock”: true, “title”: “A scripted pick-and-place”, “height”: 300}
The page's IK restarts and random draws differ from the script's, so individual episodes do not match the table below one for one. Rates over many episodes are comparable; the script's 100-episode numbers are the reference.
## Mathematics
### The state machine
| State | Target or action | Leaves when | Failure detected here |
|---|---|---|---|
| approach | tcp 15 cm above the estimated cube position, yaw matched to the cube | path finished + 0.2 s | IK failed |
| descend | tcp at 22 mm, the cube's mid-height | path finished + 0.2 s | |
| close | gripper command 0 for 0.6 s | timer | |
| check grasp | both pads touch the cube and the half-opening is in (10, 35) mm | immediately | grasp missed (after the retry) |
| lift | back to 15 cm | path finished + 0.2 s | |
| check lift | cube above 8 cm and still grasped | immediately | dropped while lifting |
| carry, lower | above the tray at 15 cm, then 7 cm | path finished + 0.2 s | dropped while carrying |
| release, retreat | open for 0.5 s, back to 15 cm, wait 0.5 s | timer | |
| judge | cube inside the tray region | | missed the tray, never picked up |
Yaw needs one subtlety. A cube looks the same every 90°, so the grasp yaw is the cube's yaw wrapped into $[-45°, 45°)$; without the wrap, the wrist would turn up to 180° for nothing and, near joint limits, fail IK.
The path between two waypoints is a straight line in joint space, not in Cartesian space. With waypoints 15 cm above the table that is harmless here; near obstacles it is not, and a Cartesian path (a sequence of IK solutions along the line, or the task-space controller of Lesson 8.3) is the fix.
### A success rate is an estimate
$k$ successes in $n$ episodes estimate the success probability $p$, and the interval says how well. With 100 of 100 the Wilson interval is [0.96, 1.00]: zero failures in 100 episodes do not show that the failure rate is zero, only that it is probably below about 4%. The one-sided version has a closed form: if the failure probability were $q$, the chance of $n$ clean runs is $(1-q)^n$, which falls below 0.05 once $q > 1 - 0.05^{1/n} \approx 3/n$. So 100 clean episodes bound the failure rate at 3%, and 30 clean episodes only at 10%.
Comparisons need the same care, and a design that makes them sharper. Here the blind and the checked runs share each episode's initial state and its first perception draw (the generator for episode $i$ is seeded with `[SEED, i]`), so the two conditions differ only where they make different decisions. A paired test then uses only the discordant states: those where one condition succeeded and the other failed. Under the null hypothesis that the check makes no difference, each discordant state is a fair coin, and the exact sign test (McNemar's exact test) gives the p-value.
### The capture region predicts the success rate
Why does a 5 mm error never matter while a 15 mm error fails a quarter of the time? Because the gripper tolerates position error inside a region around the cube, and only errors outside it fail. Output (3) measures that region at one pose by sweeping a fixed error along and across the closing axis:
- Along the closing axis, the jaws open to 80 mm around a 40 mm cube, a clearance of 20 mm per side. Inside it the closing jaws slide the cube to the centre. Every error up to 24 mm succeeds cleanly.
- Across the closing axis, the pads (16 mm wide) must overlap the cube (40 mm). Contact is geometrically possible up to $20 + 8 = 28$ mm, but every error from 22 mm on fails: 7 mm of overlap still holds, 6 mm does not, and the cube turns out of the pinch. The clean tolerance is 21 mm.
An isotropic Gaussian error stays isotropic in any rotated frame, so its components along and across the closing axis are independent with the same σ. If the capture region is roughly the rectangle $|e_\parallel| \le t_\parallel$, $|e_\perp| \le t_\perp$, the blind success probability is
$$P(\text{success}) \approx \operatorname{erf}\!\left(\frac{t_\parallel}{\sigma\sqrt 2}\right)\operatorname{erf}\!\left(\frac{t_\perp}{\sigma\sqrt 2}\right).$$
With $t_\parallel$ = 24 mm and $t_\perp$ = 21 mm this predicts 0.95 at σ = 10 mm and 0.75 at σ = 15 mm; the 100-episode runs measured 0.95 and 0.74. The region was measured at a single pose with yaw 0, yet it predicts the rate averaged over the whole 20 × 20 cm area and ±45° of yaw, which suggests the region varies little across this workspace (inference: only one pose was swept).
> [!recommendation] Measure the capture region before tuning perception
> The capture region turns a perception requirement into a number: to keep blind failures below 1% here, the error must satisfy $\operatorname{erf}(24/\sigma\sqrt2)\operatorname{erf}(21/\sigma\sqrt2) \ge 0.99$, which needs σ ≤ 7.9 mm. It also tells you where effort pays: widening the narrow direction (larger pads, a closing motion that centres the object across the jaws) buys more than improving the estimate along the wide one.
### What the contact check catches, and what it does not
The check pays where misses are common. Paired over the same 100 states at σ = 15 mm, the checked pipeline succeeded on 13 states where the blind one failed and failed on 1 where the blind one succeeded (exact sign test p = 0.0018). At σ = 10 mm the split was 4 to 0, p = 0.12: a 4-point gain that 100 paired episodes cannot separate from chance.
If the check were perfect and a second attempt independent of the first, one retry would turn a per-attempt success of 0.75 into $1 - 0.25^2 = 0.94$. At σ = 15 mm the checked runs reached 0.86. The gap has two sources in the failure list. Seven episodes are "grasp missed": both attempts fell outside the capture region. Seven are "dropped while lifting": the check passed and the cube fell. These are the check's false positives. When the pads land near the cube's edge, both touch it and the jaws stop at a plausible opening, yet the pinch holds too little of the face to carry the cube. A stricter test (an opening close to the cube's expected half-width of 19.3 mm, or a 1 cm test lift that confirms the cube follows) would catch most of them before the lift commits.
There is also a failure that looks like a success. Beyond 24 mm along the closing axis, a descending pad lands on the cube and knocks it onto another face; because every face of a cube is the same, the gripper then grasps it anyway. Output (3) shows clean successes up to 24 mm, then "success after knocking the cube over" up to 30 to 34 mm. In the noisy runs at σ = 15 mm this happened in 2 of the blind successes and 5 of the checked ones. For any object without that symmetry those episodes fail, and a criterion that looks only at the final position cannot tell them apart. Log the object's orientation during the grasp as well as where it ends up.
## Implementation
```io
INPUT: pick_place.xml; 100 random cube poses (seed 10)
PROCESS: run the state machine with the exact pose, then with Gaussian perception error, blind and with the contact check plus one retry; sweep a fixed error to measure the capture region
OUTPUT: success rates with Wilson intervals, failure categories, a paired comparison, the capture region and the predicted rates
```python file=examples/l10_2_pick_place.py “"”Lesson 10.2: a scripted pick-and-place pipeline, measured over randomized initial states.
INPUT pick_place.xml (arm7 + gripper, three cubes, a tray); the red cube is placed at a random position in a 20 x 20 cm area with a random yaw, the other cubes are set aside PROCESS a state machine over 6-D task-space waypoints: approach above the cube -> descend -> close -> check the grasp from contacts -> lift -> carry above the tray -> lower -> open -> retreat each waypoint is converted to joint angles by inverse kinematics, joints follow minimum-jerk paths tracked by PD with gravity compensation; success is judged from where the cube ends up, after the gripper has let go and moved away. (1) 100 episodes with the exact cube pose; (2) the cube’s position estimated with Gaussian error (sigma 5, 10, 15 mm per axis), run blind (no checks) and with the contact check plus one retry from a fresh estimate; (3) the capture region: one cube pose, a deterministic position error swept along and across the gripper’s closing axis, run blind; the measured half-widths then predict the blind success rates of (2) OUTPUT success rates with Wilson intervals, failure categories, what the checks caught, a paired comparison of blind and checked runs (same states, same first estimate), the capture region and the predicted blind success rates
Run: python examples/l10_2_pick_place.py “””
import math import os
import mujoco import numpy as np
from mjcourse import kinematics, model_path, spatial, stats
EPISODES = 20 if os.environ.get(“MJC_FAST”) == “1” else 100 SEED = 10 AREA = ((0.40, 0.60), (-0.20, 0.00)) # x, y ranges of the cube centre (m): 20 x 20 cm WN, H = 20.0, 0.002 # joint feedback bandwidth (rad/s), timestep (s) OPEN, CLOSED = 0.04, 0.0 # gripper half-opening commands (m)
class Episode: def init(self, cube_xy: np.ndarray, yaw: float): self.model = mujoco.MjModel.from_xml_path(str(model_path(“pick_place”))) self.data = mujoco.MjData(self.model) m, d = self.model, self.data mujoco.mj_resetDataKeyframe(m, d, m.key(“home”).id) self.arm = np.arange(7) for name, xy in ((“green_cube”, (-0.6, 0.6)), (“blue_cube”, (-0.6, -0.6))): # out of the way adr = m.jnt_qposadr[m.joint(name).id] d.qpos[adr:adr + 3] = [xy, 0.02] adr = m.jnt_qposadr[m.joint(“red_cube”).id] d.qpos[adr:adr + 7] = [cube_xy, 0.02, math.cos(yaw / 2), 0, 0, math.sin(yaw / 2)] d.ctrl[7] = OPEN mujoco.mj_forward(m, d) self.cube, self.cube_geom = m.body(“red_cube”).id, m.geom(“red_cube”).id self.pads = (m.geom(“gripper/pad_left”).id, m.geom(“gripper/pad_right”).id) self.home = d.qpos[:7].copy() self.down = spatial.mat_to_quat(d.site(“gripper/tcp”).xmat.reshape(3, 3)) # tool pointing down at home mass = np.zeros((m.nv, m.nv)) mujoco.mj_fullM(m, d, mass) self.kp = np.diag(mass)[:7] * WN**2 self.kd = 2 * np.diag(mass)[:7] * WN self.target = d.qpos[:7].copy()
def step(self, n: int = 1) -> None:
m, d = self.model, self.data
for _ in range(n):
mujoco.mj_forward(m, d)
tau = self.kp * (self.target - d.qpos[:7]) - self.kd * d.qvel[:7] + d.qfrc_bias[:7]
d.ctrl[:7] = np.clip(tau, m.actuator_ctrlrange[:7, 0], m.actuator_ctrlrange[:7, 1])
mujoco.mj_step(m, d)
def move_to(self, pos: np.ndarray, yaw: float, seconds: float) -> bool:
"""IK for the tcp pose, then a minimum-jerk joint path of the given duration."""
quat = spatial.quat_mul(np.array([math.cos(yaw / 2), 0, 0, math.sin(yaw / 2)]), self.down)
q_init = self.data.qpos.copy()
result, _ = kinematics.solve_ik_restarts(self.model, "gripper/tcp", pos, quat, q_init=q_init,
joints=list(range(7)), restarts=5, seed=0, tol_pos=5e-4, tol_rot=5e-3)
if not result.success:
return False
start, goal = self.target.copy(), result.qpos[:7].copy()
n = round(seconds / H)
for k in range(n):
s = (k + 1) / n
self.target = start + (goal - start) * (10 * s**3 - 15 * s**4 + 6 * s**5)
self.step()
self.step(round(0.2 / H)) # settle
return True
def grip(self, command: float, seconds: float = 0.6) -> None:
self.data.ctrl[7] = command
self.step(round(seconds / H))
def grasped(self) -> bool:
"""Both pads touch the cube and the jaws stopped on it, neither open nor fully closed."""
d = self.data
touching = set()
for c in d.contact[:d.ncon]:
pair = {c.geom1, c.geom2}
for pad in self.pads:
if pair == {pad, self.cube_geom}:
touching.add(pad)
opening = d.ten_length[0]
return len(touching) == 2 and 0.01 < opening < 0.035
def cube_pos(self) -> np.ndarray:
return self.data.xpos[self.cube].copy()
def tipped(self) -> bool:
"""The cube's own z axis is more than 45 degrees from vertical: it was knocked onto another face."""
return self.data.xmat[self.cube][8] < math.cos(math.pi / 4)
def attempt_grasp(ep: Episode, estimate: np.ndarray, yaw: float) -> bool: “"”Approach the estimated cube pose, descend, close; True if the move succeeded.””” return (ep.move_to(np.array([estimate, 0.15]), yaw, 1.5) and ep.move_to(np.array([estimate, 0.022]), yaw, 1.0))
def run_episode(cube_xy: np.ndarray, yaw: float, sigma: float = 0.0, check: bool = True,
retries: int = 0, rng: np.random.Generator | None = None,
bias: np.ndarray | None = None, episode: Episode | None = None) -> tuple[str, int, bool]:
“"”One pick-and-place. Returns (outcome, grasp attempts, whether the cube was knocked onto
another face before a close). With check=False nothing is verified along the way; the
outcome is judged only at the end, from the cube’s position. bias replaces the random
perception error with a fixed one; episode runs a prepared Episode (Lesson 11.3 records one).”””
ep = episode if episode is not None else Episode(cube_xy, yaw)
tray = ep.data.site(“tray_center”).xpos.copy()
grasp_yaw = (yaw + math.pi / 4) % (math.pi / 2) - math.pi / 4 # a cube looks the same every 90 degrees
attempts, tipped = 0, False
def result(outcome: str) -> tuple[str, int, bool]:
return outcome, attempts, tipped
while True:
attempts += 1
error = bias if bias is not None else (rng.normal(0.0, sigma, 2) if sigma > 0 else 0.0)
estimate = ep.cube_pos()[:2] + error # perception
if not attempt_grasp(ep, estimate, grasp_yaw):
return result("IK failed")
tipped = tipped or ep.tipped()
ep.grip(CLOSED)
if not check or ep.grasped():
break
ep.grip(OPEN, 0.4) # let go, back off, look again
ep.move_to(np.array([*estimate, 0.15]), grasp_yaw, 0.8)
if attempts > retries:
return result("grasp missed")
if not ep.move_to(np.array([*estimate, 0.15]), grasp_yaw, 1.0):
return result("IK failed")
if check and (ep.cube_pos()[2] < 0.08 or not ep.grasped()):
return result("dropped while lifting")
if not ep.move_to(np.array([tray[0], tray[1], 0.15]), 0.0, 1.5):
return result("IK failed")
if check and ep.cube_pos()[2] < 0.08:
return result("dropped while carrying")
if not ep.move_to(np.array([tray[0], tray[1], 0.07]), 0.0, 0.8):
return result("IK failed")
ep.grip(OPEN, 0.5)
ep.move_to(np.array([tray[0], tray[1], 0.15]), 0.0, 0.8)
ep.step(round(0.5 / H))
p = ep.cube_pos()
inside = abs(p[0] - tray[0]) < 0.07 and abs(p[1] - tray[1]) < 0.07 and p[2] < 0.06
if inside:
return result("success")
if np.linalg.norm(p[:2] - cube_xy) < 0.05:
return result("never picked up (found only at the end)")
return result("missed the tray")
def report(label: str, results: list[tuple[str, int, bool]], states: np.ndarray) -> None: outcomes = [o for o, _, _ in results] n = len(outcomes) wins = outcomes.count(“success”) lo, hi = stats.wilson_interval(wins, n) lucky = sum(o == “success” and t for o, _, t in results) print(f” {label:<44} success {wins:3d}/{n} = {wins / n:.2f} 95% Wilson [{lo:.2f}, {hi:.2f}]” + (f” ({lucky} only after knocking the cube onto another face)” if lucky else “”)) for category in sorted(set(outcomes) - {“success”}): idx = [i for i, o in enumerate(outcomes) if o == category] print(f” {category}: {len(idx)} e.g. (x, y, yaw deg) “ + “; “.join(f”({states[i, 0]:.3f}, {states[i, 1]:.3f}, {math.degrees(states[i, 2]):.0f})” for i in idx[:3]))
def capture_region(cube_xy: np.ndarray, max_mm: int = 40) -> dict[str, int]: “"”Sweep a fixed position error along and across the closing axis (cube yaw 0, blind) and print the outcome runs. Returns, per axis, the largest error with a clean success on both sides.””” axes = Episode(cube_xy, 0.0).data.site(“gripper/tcp”).xmat.reshape(3, 3) closing = axes[:2, 1] / np.linalg.norm(axes[:2, 1]) # the fingers slide along tcp y clean = {} for name, direction in ((“along the closing axis”, closing), (“across the closing axis”, np.array([-closing[1], closing[0]]))): limits = [] for sign in (1, -1): runs = [] for mm in range(max_mm + 1): outcome, _, tipped = run_episode(cube_xy, 0.0, check=False, bias=sign * mm / 1000 * direction) label = “success after knocking the cube over” if outcome == “success” and tipped else outcome if runs and runs[-1][0] == label: runs[-1][2] = mm else: runs.append([label, mm, mm]) limits.append(runs[0][2] if runs[0][0] == “success” else -1) print(f” {name}, {‘+’ if sign > 0 else ‘-‘}: “ + “; “.join(f”{a}-{b} mm {lab}” for lab, a, b in runs)) clean[name] = min(limits) return clean
def sign_test(wins: int, losses: int) -> float: “"”Exact two-sided p-value for discordant pairs (McNemar’s exact test).””” n = wins + losses if n == 0: return 1.0 tail = sum(math.comb(n, k) for k in range(min(wins, losses) + 1)) / 2**n return min(1.0, 2 * tail)
def two_sided(t: float, sigma: float) -> float: “"”P(|e| <= t) for e ~ N(0, sigma^2).””” return math.erf(t / (sigma * math.sqrt(2)))
if name == “main”: rng = np.random.default_rng(SEED) states = np.column_stack([rng.uniform(AREA[0], EPISODES), rng.uniform(AREA[1], EPISODES), rng.uniform(-math.pi / 4, math.pi / 4, EPISODES)]) print(f”{EPISODES} initial states (seed {SEED}): red cube in x {AREA[0]}, y {AREA[1]} m, yaw +-45 deg, MuJoCo {mujoco.version}”) print(“(1) exact cube pose”) report(“contact checks on”, [run_episode(s[:2], s[2]) for s in states], states) print(“(2) cube position estimated with error (Gaussian, per axis)”) blind_rates = {} for sigma in (0.005, 0.010, 0.015): blind = [run_episode(s[:2], s[2], sigma, check=False, rng=np.random.default_rng([SEED, i])) for i, s in enumerate(states)] checked = [run_episode(s[:2], s[2], sigma, check=True, retries=1, rng=np.random.default_rng([SEED, i])) for i, s in enumerate(states)] report(f”sigma {1000 * sigma:.0f} mm, blind”, blind, states) report(f”sigma {1000 * sigma:.0f} mm, contact check + 1 retry”, checked, states) print(f” retries used: {sum(a > 1 for _, a, _ in checked)} episodes”) gains = sum(c[0] == “success” and b[0] != “success” for b, c in zip(blind, checked)) losses = sum(b[0] == “success” and c[0] != “success” for b, c in zip(blind, checked)) print(f” paired: checked succeeded where blind failed on {gains} states, the reverse on {losses}; “ f”exact sign test p = {sign_test(gains, losses):.2g}”) blind_rates[sigma] = sum(o == “success” for o, _, _ in blind) / len(blind) print(“(3) capture region at cube (0.50, -0.10), yaw 0, blind, fixed position error”) clean = capture_region(np.array([0.50, -0.10])) t_close, t_across = clean[“along the closing axis”], clean[“across the closing axis”] print(f” clean capture half-widths: {t_close} mm along, {t_across} mm across the closing axis”) print(“ predicted blind success P(|e_along| <= t_along) P(|e_across| <= t_across) vs measured in (2)”) for sigma, measured in blind_rates.items(): p = two_sided(t_close / 1000, sigma) * two_sided(t_across / 1000, sigma) print(f” sigma {1000 * sigma:.0f} mm: predicted {p:.2f}, measured {measured:.2f}”)
Output:
```text
100 initial states (seed 10): red cube in x (0.4, 0.6), y (-0.2, 0.0) m, yaw +-45 deg, MuJoCo 3.14.0
(1) exact cube pose
contact checks on success 100/100 = 1.00 95% Wilson [0.96, 1.00]
(2) cube position estimated with error (Gaussian, per axis)
sigma 5 mm, blind success 100/100 = 1.00 95% Wilson [0.96, 1.00]
sigma 5 mm, contact check + 1 retry success 100/100 = 1.00 95% Wilson [0.96, 1.00]
retries used: 0 episodes
paired: checked succeeded where blind failed on 0 states, the reverse on 0; exact sign test p = 1
sigma 10 mm, blind success 95/100 = 0.95 95% Wilson [0.89, 0.98]
missed the tray: 1 e.g. (x, y, yaw deg) (0.404, -0.130, 1)
never picked up (found only at the end): 4 e.g. (x, y, yaw deg) (0.430, -0.008, -9); (0.569, -0.163, -25); (0.535, -0.185, -41)
sigma 10 mm, contact check + 1 retry success 99/100 = 0.99 95% Wilson [0.95, 1.00]
grasp missed: 1 e.g. (x, y, yaw deg) (0.569, -0.163, -25)
retries used: 6 episodes
paired: checked succeeded where blind failed on 4 states, the reverse on 0; exact sign test p = 0.12
sigma 15 mm, blind success 74/100 = 0.74 95% Wilson [0.65, 0.82] (2 only after knocking the cube onto another face)
missed the tray: 2 e.g. (x, y, yaw deg) (0.445, -0.077, -28); (0.450, -0.163, -38)
never picked up (found only at the end): 24 e.g. (x, y, yaw deg) (0.430, -0.111, -18); (0.427, -0.038, -6); (0.515, -0.039, -33)
sigma 15 mm, contact check + 1 retry success 86/100 = 0.86 95% Wilson [0.78, 0.91] (5 only after knocking the cube onto another face)
dropped while lifting: 7 e.g. (x, y, yaw deg) (0.430, -0.111, -18); (0.429, -0.091, -30); (0.549, -0.007, 36)
grasp missed: 7 e.g. (x, y, yaw deg) (0.581, -0.109, 30); (0.450, -0.163, -38); (0.430, -0.027, 22)
retries used: 25 episodes
paired: checked succeeded where blind failed on 13 states, the reverse on 1; exact sign test p = 0.0018
(3) capture region at cube (0.50, -0.10), yaw 0, blind, fixed position error
along the closing axis, +: 0-24 mm success; 25-34 mm success after knocking the cube over; 35-40 mm never picked up (found only at the end)
along the closing axis, -: 0-24 mm success; 25-30 mm success after knocking the cube over; 31-31 mm success; 32-40 mm never picked up (found only at the end)
across the closing axis, +: 0-21 mm success; 22-40 mm never picked up (found only at the end)
across the closing axis, -: 0-21 mm success; 22-40 mm never picked up (found only at the end)
clean capture half-widths: 24 mm along, 21 mm across the closing axis
predicted blind success P(|e_along| <= t_along) P(|e_across| <= t_across) vs measured in (2)
sigma 5 mm: predicted 1.00, measured 1.00
sigma 10 mm: predicted 0.95, measured 0.95
sigma 15 mm: predicted 0.75, measured 0.74
The run takes about three minutes on one CPU core (MuJoCo 3.14.0). Set MJC_FAST=1 for a 20-episode version that the test suite uses.
Every episode fails at the same state with “IK failed”. The target is outside the workspace or its orientation is unreachable within the joint limits. Check the target against the arm’s reach, wrap the yaw (a cube’s grasp yaw never needs more than 45°), and use restarts: clamping at joint limits creates local minima that a different start escapes.
The pipeline reports success, but the video shows the gripper closing on air. The success criterion reads the plan (the arm reached the tray) instead of the world (the cube is in the tray). Judge from the object’s position after the gripper has let go and moved away, as here.
Grasps pass the check and then drop on the lift. Edge pinches: both pads touch, the opening looks plausible, the hold is weak. Narrow the accepted opening band around the expected object width, or add a short test lift.
Successes that should not count. The object was knocked over or pushed into a graspable pose. Record its orientation and position at grasp time and flag large changes from the initial state.
Two runs of the same evaluation give different rates. Unseeded randomness. Seed each episode’s generator from the evaluation seed and the episode index, so that conditions compared on the same seed are paired and a rerun reproduces every episode.
Using the capture half-widths of output (3), predict the blind success rate at σ = 20 mm. Then add 0.020 to the list of σ values in the script, run it, and check that the measurement falls in the Wilson interval around your prediction’s corresponding count.
Make the grasp check catch edge pinches. Design a stricter test (for example, accept the grasp only if the half-opening is within 3 mm of 19.3 mm and the cube follows a 1 cm test lift), run it on the same 100 paired states at σ = 15 mm, and report both error rates of the check: grasps it rejected that would have held, and grasps it accepted that dropped. Compare the end-to-end success rate with this lesson’s 0.86 using the paired sign test.
[!research] Evaluation hygiene for manipulation policies Learned manipulation policies are often compared on a few dozen episodes per task. With 50 episodes, a measured success of 0.80 carries a 95% Wilson interval of [0.67, 0.89]: differences of several points between two methods are not resolved, and a 0.05 half-width would need about 250 episodes. Pairing the conditions (same initial states, same seeds) and testing on discordant episodes is the cheap way to sharpen a comparison. Two further lessons carry over from this scripted pipeline (inference). Final-state success criteria can count lucky successes, such as an object knocked into a graspable pose, so the reported rate can exceed what the policy reliably does. And scripted pipelines like this one are the standard source of demonstrations for imitation learning (Level 14): filtering episodes by final success removes the “never picked up” failures but keeps the lucky ones, which then teach the policy to knock objects over.
{"id": "10.2-check", "title": "Knowledge check", "questions": [
{"kind": "numeric", "q": "A pipeline succeeds in 60 of 60 episodes. Using the one-sided bound 1 − 0.05^(1/n), what failure rate can you exclude at 95% confidence?",
"answer": 0.049, "tol": 0.002,
"explain": "<p>$1 - 0.05^{1/60}$ = 0.049, close to the rule of thumb $3/n$ = 0.05. Sixty clean episodes say the failure rate is probably below 5%, not that it is zero.</p>"},
{"kind": "numeric", "q": "Capture half-widths are 24 mm along and 21 mm across the closing axis. Predict the blind success rate for an isotropic Gaussian position error with σ = 10 mm per axis.",
"answer": 0.95, "tol": 0.01,
"explain": "<p>$\\operatorname{erf}(24/(10\\sqrt2))\\,\\operatorname{erf}(21/(10\\sqrt2))$ = 0.984 × 0.964 = 0.95, which output (2) measured.</p>"},
{"kind": "mcq", "q": "At σ = 15 mm a single attempt succeeds about 75% of the time. A perfect check with one independent retry would give about 94%, but the checked runs reached 86%. What accounts for most of the gap?",
"options": ["The retry draws the same perception error", "Edge pinches pass the contact check and then drop on the lift, together with episodes that miss twice", "The IK fails more often on retries", "The contact check is too slow"],
"answer": 1,
"explain": "<p>Output (2): 7 drops while lifting (false positives of the check) and 7 double misses.</p>"},
{"kind": "mcq", "q": "Why is a success that follows a pad knocking the cube onto another face a problem for evaluation?",
"options": ["It is not a problem: the cube reached the tray", "It depends on the cube's symmetry; the same episode fails for most objects, and a final-position criterion cannot see the difference", "It breaks the Wilson interval", "MuJoCo cannot simulate tipping accurately"],
"answer": 1,
"explain": "<p>The final state is identical to a clean success, so the rate overstates how reliably the pipeline grasps. Logging orientation during the grasp exposes it.</p>"}
]}
Lesson 10.3 leaves grasping for tasks whose difficulty is the contact itself: pushing an object to a goal, inserting a peg with 2 mm of clearance under impedance control, and opening a drawer and a door.