Domain randomization tunes or trains a controller on a distribution of simulator parameters instead of one value, so that the real system, whose parameters are known only roughly, looks like one more draw from that distribution. Whether it helps depends on three things, and each needs a measurement rather than an assumption. The parameter change has to reach the simulation correctly. The comparison has to be against a controller tuned on nominal values, evaluated on held-out values that reach beyond the training range. And the outcome has to depend on the randomized parameter under the controller being tuned. This lesson tunes a pushing controller on nominal and on randomized puck mass and friction and evaluates both on a 5 × 5 grid of held-out values. It then repeats the comparison for an open-loop strike, whose outcome depends on friction directly, and adds a third option: measuring the friction before acting.
[!established] What is randomized, and how
PushEnv.reset(options=...)accepts"puck_mass"(kg; the inertia is scaled with the mass,mj_setConstrecomputes the derived constants, and the state is reset) and"friction"(the sliding coefficient of both the puck and the floor). At equal priority MuJoCo gives a contact the larger of the two geoms’ coefficients (measured here: puck 1.0 on floor 0.25 gives a contact friction of 1.0, and so does the reverse), so setting both sets the contact’s. Values not given return to nominal: 0.2 kg and 0.5. Training draws mass from 0.1 to 0.4 kg and friction from 0.25 to 1.0 in every episode. The held-out grid is mass 0.05, 0.1, 0.2, 0.4, 0.8 kg by friction 0.1, 0.25, 0.5, 1.0, 1.5, with the same 24 tasks in every cell for every controller.
The lab strikes a puck toward a goal 10 to 25 cm away by launching it at $v = \sqrt{2\hat\mu g D}$ and letting it slide. Success means it stops within 3 cm of the goal. Each strategy picks $\hat\mu$ differently; Sweep the true friction plots its success over 20 tasks against the friction the puck actually has.
```lab dr {“dock”: true, “title”: “Randomize, or measure?”}
The lab's scene is the puck alone, launched along one axis; the script below uses the full push scene with the arm and two-dimensional goals, so its numbers differ in detail but not in pattern.
## Mathematics
### (0) A mass change that does not reach the dynamics
Editing `model.body_mass` and `model.body_inertia` changes two fields, but MuJoCo also stores constants derived from them, computed by `mj_setConst` when the model is compiled. After doubling the puck's mass, `mj_setConst` changes six fields: `body_invweight0`, `dof_invweight0`, `dof_M0`, `body_subtreemass`, `cam_poscom0` and `light_poscom0` (the last two are reference positions for cameras and lights that track a centre of mass). Without that call, the push episodes diverge:
| Puck mass | Diverged, edited without `mj_setConst` | Diverged, through `reset(options={"puck_mass": m})` |
|---|---|---|
| 0.2 kg (nominal) | 0/10 | 0/10 |
| 0.4 kg | 10/10 | 0/10 |
| 0.8 kg | 10/10 | 0/10 |
Recomputing one field at a time at 0.4 kg isolates the cause: with only `body_subtreemass` recomputed no episode diverges (0/10), with only `dof_M0` 8/10 diverge, and with only either inverse weight all 10 do. Why a stale subtree mass destabilizes this step was not traced here; the rule does not depend on it. Call `mj_setConst` after editing any inertial parameter, then reset. MuJoCo reports the divergence as a warning (printed to stderr and counted in `data.warning`) and resets the state, time included, so a randomized training run with stale constants does not crash. It quietly trains on episodes that restart, which is why the script detects divergence by the clock running backwards.
### (1) to (3) A closed-loop pusher: randomization buys nothing measurable
The pusher has four parameters: push speed, a gain that slows it near the goal ($\min(v, k_\text{slow} d)$), push depth, and the lateral error it tolerates before repositioning behind the puck. The cross-entropy method tunes them on nominal dynamics and, separately, with mass and friction randomized in every episode (24 candidates, 8 episodes per candidate, 12 iterations). Nominal tuning ends at mean success 0.99 and randomized tuning at 0.88, where every candidate scored 7 of 8. The two parameter sets differ (push speed 0.107 against 0.181 m/s), and on the held-out grid both succeed almost everywhere:
| Mean success | Inside the randomized range | Outside it | Nominal cell |
|---|---|---|---|
| tuned on nominal | 1.00 | 0.99 | 1.00 |
| tuned with randomization | 1.00 | 1.00 | 1.00 |
Over all 25 cells the nominal-tuned pusher fails 3 of 600 episodes (2 of them outside the range) and the randomized one fails 1 (inside it); single failures in cells of 24 tasks are too few to call a difference. The reason is physical. The pusher commands positions every 50 ms and the Cartesian impedance can apply up to about 40 N (2000 N/m times the 2 cm cap on how far the target may lead the tool). Sliding the heaviest, grippiest puck on the grid takes $\mu m g$ = 1.5 × 0.8 × 9.81 = 11.8 N, so the pusher is never force-limited, and the puck moves with it. Near the goal the speed is $k_\text{slow} d$, so the puck coasts a negligible distance when the pusher stops: from $d$ = 1 cm at the nominal gain, $v$ = 0.022 m/s and the coast at $\mu$ = 0.1 is $v^2 / 2\mu g \approx$ 0.24 mm. Feedback and a force margin make the outcome insensitive to mass and friction, so randomizing them has nothing to fix.
> [!recommendation] Measure sensitivity before randomizing
> Evaluate the nominal controller over held-out parameter values first. Where its success is flat, randomization has nothing to buy and costs training time (here the randomized tuning also scored lower during training, 0.88 against 0.99). Spend randomization on the parameters the outcome depends on, and check that every value in the range leaves the task feasible: here the largest sliding force on the grid, 11.8 N, is well below the 40 N the pusher can apply. A range that crosses a feasibility limit changes the task instead of the dynamics.
### (4) An open-loop strike: randomization moves the window
The strike launches the puck at $v = \sqrt{2\hat\mu g D}$ for a goal $D$ away and lets it slide. Under Coulomb friction the puck slides $v^2/(2\mu g) = D\,\hat\mu/\mu$, so it stops within the success radius $r$ = 3 cm only if
$$\left|\frac{\hat\mu}{\mu} - 1\right| \le \frac{r}{D}.$$
PushEnv's goals are 8 to 30 cm from the puck, so the tolerance on $\hat\mu/\mu$ is ±10% to ±37%, about ±20% at 15 cm. The simulator follows the formula: after a launch at 0.4 m/s the puck slides 81.2, 32.4, 16.1 and 8.2 mm at $\mu$ = 0.1, 0.25, 0.5 and 1.0, against 81.5, 32.6, 16.3 and 8.2 mm predicted.
$\hat\mu$ is chosen by grid search (0.05 to 1.5 in steps of 0.05) on 64 training tasks, once at the nominal friction and once with friction drawn from the randomized range. Nominal tuning picks 0.45 with a training success of 1.00 (the search keeps the first value that reaches the best score). Randomized tuning picks 0.7, the best any single value can do over 0.25 to 1.0, and that best is a training success of 0.44. On 24 held-out tasks per friction:
| True friction | 0.05 | 0.1 | 0.25 | 0.5 | 0.75 | 1.0 | 1.25 | 1.5 |
|---|---|---|---|---|---|---|---|---|
| $\hat\mu$ 0.45 (chosen on nominal) | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| $\hat\mu$ 0.7 (chosen on randomized) | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 |
| probe at 0.3 m/s, then strike | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.96 | 0.71 |
| oracle: $\hat\mu$ = true $\mu$ | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.54 |
| its largest tilt, median (degrees) | 0.1 | 0.2 | 0.2 | 0.6 | 1.3 | 1.1 | 3.9 | 116.1 |
Both single-value strikes succeed at one friction each; inside the randomized range their mean success is the same, 0.25. A controller that cannot observe the friction has one operating point, and training it on a distribution only chooses where that point sits. What extends the window is information about the parameter.
### Identification by a probe, and its operating range
The probe launches the puck at a known speed $v_0$, waits 1 s, and estimates $\hat\mu = v_0^2/(2gd)$ from the distance $d$ slid. Its accuracy depends on how far the puck slides:
| Probe speed | true 0.05 | 0.25 | 0.5 | 1.0 | 1.5 |
|---|---|---|---|---|---|
| 0.15 m/s: slide, estimate | 22.8 mm, 0.05 | 4.5 mm, 0.25 | 2.5 mm, 0.46 | 1.5 mm, 0.77 | 1.4 mm, 0.80 |
| 0.3 m/s: slide, estimate | 91.5 mm, 0.05 | 18.1 mm, 0.25 | 9.2 mm, 0.50 | 4.7 mm, 0.97 | 3.2 mm, 1.45 |
At 0.15 m/s and $\mu$ = 1.5 the formula predicts a 0.76 mm slide and the simulation gives 1.4 mm; at slides of a millimetre or two the extra distance dominates and the estimate saturates near 0.8. Doubling the probe speed quadruples the slide and keeps the estimate within 4% up to 1.5. Every measurement has an operating range like this, and it has to be measured, not assumed.
The last rows of the strike table separate estimation error from model error. Told the true friction, the strike succeeds at every friction up to 1.25 and at 0.54 at 1.5, below the probe's 0.71, so the failure at 1.5 does not come from the estimate. The puck stops sliding: the median of its largest tilt during a strike is 3.9° at 1.25 and 116° at 1.5. At the highest friction it tips over and tumbles, and $v^2/(2\mu g)$ no longer describes it. The physics check above stopped at 1.0; a model's range of validity needs the same check as an estimator's.
> [!recommendation] Randomize, adapt, or identify
> Randomization makes a controller insensitive to a parameter only if some single behaviour works across the range. Feedback can provide that (the pusher), an open-loop action cannot (the strike). When the outcome depends on a parameter the controller cannot observe, give it information: identify the parameter (the probe; Level 18), or let the policy infer it from its own history while it acts. Randomization is still needed for the second, because the policy has to see the range to learn how to infer within it.
## Implementation
```io
INPUT: PushEnv with puck mass and friction set at reset; a parametric pusher; an open-loop strike
PROCESS: (0) mass edits with and without mj_setConst; (1, 2) CEM tuning on nominal and on randomized dynamics; (3) a held-out 5 x 5 grid; (4) strikes with friction tuned, randomized, probed, and known
OUTPUT: printed tables
```python file=examples/l17_1_domain_randomization.py “"”Lesson 17.1: domain randomization for the push task, measured against held-out parameters (Level 17 checkpoint).
INPUT mjcourse.envs.PushEnv (end-effector actions whose target leads the tool by at most 2 cm, episodes of 5 s, success judged where the puck ends) PROCESS (0) a pitfall: the puck’s mass changed in the model with and without mj_setConst, and with one derived field at a time recomputed; a parametric pushing controller (push speed, slowdown gain near the goal, push depth, lateral tolerance) tuned by the cross-entropy method twice: (1) on nominal dynamics (puck 0.2 kg, friction 0.5); (2) with puck mass drawn from 0.1-0.4 kg and friction from 0.25-1.0 in every episode; (3) both evaluated on a grid of mass and friction values reaching beyond the randomized range, 24 tasks per cell, the same tasks for both controllers; (4) an open-loop strike: the puck is launched toward the goal at v = sqrt(2 mu_hat g D) for its distance D and an assumed friction mu_hat, then slides; mu_hat chosen on nominal friction and on the randomized range; success against the true friction; first, the simulator’s sliding distance against v^2 / (2 mu g); then a strike that first identifies the friction from a probe launch, after checking the probe’s estimates; last, a strike told the true friction, which separates estimation error from model error OUTPUT printed tables
Run: python examples/l17_1_domain_randomization.py (about 3 minutes on 24 CPU cores) “””
import math import os from multiprocessing import Pool
import mujoco import numpy as np
from mjcourse.envs import PushEnv
FAST = os.environ.get(“MJC_FAST”) == “1” R_PUCK, HALF, DT = 0.035, 0.008, 0.05 NOMINAL = {“puck_mass”: 0.2, “friction”: 0.5} TRAIN_RANGE = {“puck_mass”: (0.1, 0.4), “friction”: (0.25, 1.0)} GRID = {“puck_mass”: (0.05, 0.1, 0.2, 0.4, 0.8), “friction”: (0.1, 0.25, 0.5, 1.0, 1.5)}
def decode(theta: np.ndarray) -> dict: s = 1 / (1 + np.exp(-np.asarray(theta))) return {“speed”: 0.05 + 0.25 * s[0], “slow_gain”: 0.5 + 4.5 * s[1], “depth”: 0.002 + 0.02 * s[2], “lateral”: 0.005 + 0.03 * s[3]}
def pusher(p: dict, obs: np.ndarray) -> np.ndarray: “"”Get behind the puck on its line to the goal, then push along that line, slowing near the goal.””” tcp, puck, goal = obs[14:16], obs[17:19], obs[19:21] to_goal = goal - puck d = np.linalg.norm(to_goal) if d < 0.01: return np.zeros(2, np.float32) u = to_goal / d n = np.array([-u[1], u[0]]) rel = tcp - puck along, lateral = rel @ u, rel @ n contact = -(R_PUCK + HALF) if along > contact + 0.015 or abs(lateral) > p[“lateral”]: # not behind the puck: reposition side = n * math.copysign(R_PUCK + HALF + 0.03, lateral if lateral != 0 else 1.0) target = puck + side if along > contact else puck - (R_PUCK + HALF + 0.03) * u speed = 0.3 else: # push target = puck - (R_PUCK + HALF - p[“depth”]) * u speed = min(p[“speed”], p[“slow_gain”] * d) step = target - tcp dist = np.linalg.norm(step) move = min(speed * DT, dist) return np.clip(step / max(dist, 1e-9) * move / 0.02, -1, 1).astype(np.float32)
_ENV = {}
def episode(job) -> float: “"”One episode: (theta, task seed, mass, friction) -> 1 success, 0 failure, nan if the simulation diverged. MuJoCo resets a diverging simulation to its initial state (time 0), so a clock that runs backwards, or a controller that fails at the reset pose, marks the episode.””” theta, seed, mass, friction = job if “pusher” not in _ENV: _ENV[“pusher”] = PushEnv(action_mode=”ee_delta”, terminate_on_success=False, max_lead=0.02) # force at most about 40 N env, p = _ENV[“pusher”], decode(theta) obs, _ = env.reset(seed=int(seed), options={“puck_mass”: mass, “friction”: friction}) done, last_time = False, 0.0 try: while not done: obs, _, terminated, truncated, info = env.step(pusher(p, obs)) if env.data.time < last_time: return float(“nan”) last_time, done = env.data.time, terminated or truncated except np.linalg.LinAlgError: return float(“nan”) return float(info[“distance”] < env.success_radius)
STALE = (“body_invweight0”, “dof_invweight0”, “dof_M0”, “body_subtreemass”) # what mj_setConst changes after a mass edit
def stale_constants(mass: float, episodes: int, repair=None) -> int: “"”Diverged episodes when the puck’s mass and inertia are edited directly. repair=None takes the environment’s path (edit, mj_setConst, reset); otherwise only the listed derived fields are recomputed.””” env = PushEnv(action_mode=”ee_delta”, terminate_on_success=False, max_lead=0.02) m, puck, diverged = env.model, env.puck, 0 nominal_mass, nominal_inertia = m.body_mass[puck], m.body_inertia[puck].copy() m.body_mass[puck], m.body_inertia[puck] = mass, nominal_inertia * mass / nominal_mass mujoco.mj_setConst(m, env.data) correct = {k: getattr(m, k).copy() for k in STALE} # the constants for the edited mass m.body_mass[puck], m.body_inertia[puck] = nominal_mass, nominal_inertia mujoco.mj_setConst(m, env.data) for seed in range(episodes): if repair is None: obs, _ = env.reset(seed=seed, options={“puck_mass”: mass}) else: # the same edit made directly obs, _ = env.reset(seed=seed) m.body_mass[puck], m.body_inertia[puck] = mass, nominal_inertia * mass / nominal_mass for k in repair: getattr(m, k)[:] = correct[k] try: last, done = 0.0, False while not done: obs, _, terminated, truncated, _ = env.step(pusher(decode(np.zeros(4)), obs)) if env.data.time < last: diverged += 1 break last, done = env.data.time, terminated or truncated except np.linalg.LinAlgError: diverged += 1 m.body_mass[puck], m.body_inertia[puck] = nominal_mass, nominal_inertia mujoco.mj_setConst(m, env.data) # leave the model as it was env.close() return diverged
def cem(pool, randomize: bool, seed: int, iterations: int, population: int, episodes: int) -> np.ndarray: rng = np.random.default_rng(seed) mean, std = np.zeros(4), np.full(4, 1.5) for it in range(iterations): thetas = rng.normal(mean, std, (population, 4)) tasks = rng.integers(0, 10*6, episodes) # shared tasks within an iteration if randomize: params = [(rng.uniform(TRAIN_RANGE[“puck_mass”]), rng.uniform(*TRAIN_RANGE[“friction”])) for _ in range(episodes)] else: params = [(NOMINAL[“puck_mass”], NOMINAL[“friction”])] * episodes jobs = [(th, t, m, f) for th in thetas for t, (m, f) in zip(tasks, params)] scores = np.nan_to_num(np.array(pool.map(episode, jobs))).reshape(population, episodes).mean(axis=1) elite = thetas[np.argsort(scores)[-max(2, population // 5):]] mean, std = elite.mean(axis=0), elite.std(axis=0) + 0.05 if it in (0, iterations - 1): print(f” iteration {it + 1:>2}: mean success {scores.mean():.2f}, best {scores.max():.2f}”) return mean
def strike(job) -> tuple[float, float, float]: “"”Launch the puck toward the goal at sqrt(2 mu_hat g D) and let it slide for 3 s with the arm holding its start pose: (final distance to the goal, distance the puck travelled, largest tilt of its axis from vertical in degrees, sampled every 50 ms).””” mu_hat, seed, friction = job if “strike” not in _ENV: _ENV[“strike”] = PushEnv(action_mode=”ee_delta”, terminate_on_success=False, max_episode_steps=60) env = _ENV[“strike”] env.reset(seed=int(seed), options={“friction”: friction}) start, goal = env.puck_xy(), env.goal_xy() distance = np.linalg.norm(goal - start) dof = env.model.jnt_dofadr[env.model.joint(“puck”).id] env.data.qvel[dof:dof + 2] = math.sqrt(2 * mu_hat * 9.81 * distance) * (goal - start) / distance tilt = 0.0 for _ in range(env.max_episode_steps): env.step(np.zeros(2, np.float32)) tilt = max(tilt, math.degrees(math.acos(np.clip(env.data.xmat[env.puck][8], -1.0, 1.0)))) return float(np.linalg.norm(env.puck_xy() - goal)), float(np.linalg.norm(env.puck_xy() - start)), tilt
PROBE = 0.3 # probe launch speed (m/s)
def probe_estimate(speed: float, friction: float, seed: int = 3) -> tuple[float, float]:
“"”Launch toward the goal at speed, wait 1 s: (distance slid, friction estimated as v^2 / (2 g d)).”””
if “strike” not in _ENV:
_ENV[“strike”] = PushEnv(action_mode=”ee_delta”, terminate_on_success=False, max_episode_steps=60)
env = _ENV[“strike”]
env.reset(seed=seed, options={“friction”: friction})
dof = env.model.jnt_dofadr[env.model.joint(“puck”).id]
start, goal = env.puck_xy(), env.goal_xy()
env.data.qvel[dof:dof + 2] = speed * (goal - start) / np.linalg.norm(goal - start)
for _ in range(20):
env.step(np.zeros(2, np.float32))
slid = float(np.linalg.norm(env.puck_xy() - start))
return slid, speed**2 / (2 * 9.81 * max(slid, 1e-4))
def probe_then_strike(job) -> float: “"”Launch at PROBE m/s, wait 1 s, estimate mu from the slide (v0^2 / (2 g d)), then strike from where the puck stopped. Returns the final distance to the goal.””” _, seed, friction = job if “strike” not in _ENV: _ENV[“strike”] = PushEnv(action_mode=”ee_delta”, terminate_on_success=False, max_episode_steps=60) env = _ENV[“strike”] env.reset(seed=int(seed), options={“friction”: friction}) dof = env.model.jnt_dofadr[env.model.joint(“puck”).id]
def launch(speed: float, steps: int) -> float:
start, goal = env.puck_xy(), env.goal_xy()
env.data.qvel[dof:dof + 2] = speed * (goal - start) / np.linalg.norm(goal - start)
for _ in range(steps):
env.step(np.zeros(2, np.float32))
return float(np.linalg.norm(env.puck_xy() - start))
slid = launch(PROBE, 20)
mu_hat = PROBE**2 / (2 * 9.81 * max(slid, 1e-4))
launch(math.sqrt(2 * mu_hat * 9.81 * np.linalg.norm(env.goal_xy() - env.puck_xy())), 40)
return float(np.linalg.norm(env.puck_xy() - env.goal_xy()))
MU_GRID = (0.05, 0.1, 0.25, 0.5, 0.75, 1.0, 1.25, 1.5)
def strike_success(pool, mu_hat: float, frictions, tasks) -> float: jobs = [(mu_hat, t, f) for f, t in zip(frictions, tasks)] return float(np.mean([d < 0.03 for d, _, _ in pool.map(strike, jobs)]))
def part4(pool, per_cell: int) -> dict: “"”The strike: physics check, then mu_hat chosen by grid search on nominal and on randomized friction.””” print(“ sliding distance after a launch at 0.4 m/s: simulated against v^2 / (2 mu g)”) for mu in (0.1, 0.25, 0.5, 1.0): # mu_hat chosen so that the launch speed is 0.4 m/s for this task’s distance env = PushEnv(action_mode=”ee_delta”, max_episode_steps=60) env.reset(seed=3) d = np.linalg.norm(env.goal_xy() - env.puck_xy()) env.close() _, travelled, _ = strike((0.16 / (2 * 9.81 * d), 3, mu)) print(f” mu {mu:<5} simulated {1000 * travelled:6.1f} mm, predicted {1000 * 0.16 / (2 * mu * 9.81):6.1f} mm”) print(“ friction estimated by a probe launch (one task), at two probe speeds:”) for speed in (0.15, PROBE): cells = [] for mu in (0.05, 0.25, 0.5, 1.0, 1.5): slid, mu_hat = probe_estimate(speed, mu) cells.append(f”{mu}: {1000 * slid:.1f} mm -> {mu_hat:.2f}”) print(f” {speed} m/s “ + “; “.join(cells)) rng = np.random.default_rng(17) train_tasks = rng.integers(0, 106, 64) candidates = np.round(np.arange(0.05, 1.51, 0.05), 2) chosen = {} for name, frictions in ((“nominal”, np.full(64, NOMINAL[“friction”])), (“randomized”, rng.uniform(*TRAIN_RANGE[“friction”], 64))): scores = [strike_success(pool, c, frictions, train_tasks) for c in candidates] chosen[name] = float(candidates[int(np.argmax(scores))]) print(f” chosen on {name} friction: mu_hat {chosen[name]} (training success {max(scores):.2f})”) tasks = np.random.default_rng(1718).integers(0, 106, per_cell) print(f” success against the true friction, {per_cell} held-out tasks each:”) print(f” {‘true friction’:<34}” + ““.join(f”{m:>7}” for m in MU_GRID)) table = {} for name, mu_hat in chosen.items(): table[name] = [strike_success(pool, mu_hat, np.full(per_cell, m), tasks) for m in MU_GRID] print(f” {f’mu_hat {mu_hat} (chosen on {name})’:<34}” + ““.join(f”{v:>7.2f}” for v in table[name])) table[“probe”] = [float(np.mean([d < 0.03 for d in pool.map(probe_then_strike, [(0, t, m) for t in tasks])])) for m in MU_GRID] print(f” {f’probe at {PROBE} m/s, then strike’:<34}” + ““.join(f”{v:>7.2f}” for v in table[“probe”])) oracle = [pool.map(strike, [(m, t, m) for t in tasks]) for m in MU_GRID] print(f” {‘oracle: mu_hat = true mu’:<34}” + ““.join(f”{np.mean([d < 0.03 for d, _, _ in o]):>7.2f}” for o in oracle)) print(f” {‘ its largest tilt (deg, median)’:<34}” + ““.join(f”{np.median([a for _, _, a in o]):>7.1f}” for o in oracle)) return table
def grid(pool, theta, tasks) -> tuple[np.ndarray, np.ndarray]: “"”Success per cell (diverged episodes count as failures) and the number of diverged episodes per cell.””” jobs = [(theta, t, m, f) for m in GRID[“puck_mass”] for f in GRID[“friction”] for t in tasks] out = np.array(pool.map(episode, jobs)).reshape(len(GRID[“puck_mass”]), len(GRID[“friction”]), len(tasks)) return np.nan_to_num(out).mean(axis=2), np.isnan(out).sum(axis=2)
def show(table: np.ndarray, fmt: str = “”) -> None: print(“ mass \ friction “ + ““.join(f”{f:>7}” for f in GRID[“friction”])) for m, row in zip(GRID[“puck_mass”], table): print(f” {m:>8.2f} kg “ + ““.join(fmt.format(v) for v in row))
def inside(m: float, f: float) -> bool: return TRAIN_RANGE[“puck_mass”][0] <= m <= TRAIN_RANGE[“puck_mass”][1] and TRAIN_RANGE[“friction”][0] <= f <= TRAIN_RANGE[“friction”][1]
if name == “main”: iterations, population, episodes, per_cell = (2, 8, 2, 4) if FAST else (12, 24, 8, 24) print(f”CEM: {population} candidates x {episodes} episodes per iteration, {iterations} iterations; “ f”randomized range: mass {TRAIN_RANGE[‘puck_mass’]} kg, friction {TRAIN_RANGE[‘friction’]}”) n0 = 3 if FAST else 10 print(f”(0) puck mass edited in the model, {n0} episodes each: diverged episodes”) for mass in (0.2, 0.4, 0.8): print(f” {mass} kg: without mj_setConst {stale_constants(mass, n0, ())}/{n0}, with it {stale_constants(mass, n0)}/{n0}”) print(“ 0.4 kg, without mj_setConst but with one derived field recomputed: “ + “, “.join(f”{k} {stale_constants(0.4, n0, (k,))}/{n0}” for k in STALE)) with Pool(8 if FAST else 24) as pool: print(“(1) tuned on nominal dynamics”) theta_nominal = cem(pool, False, 0, iterations, population, episodes) print(“(2) tuned with randomized mass and friction”) theta_random = cem(pool, True, 0, iterations, population, episodes) for name, th in ((“nominal”, theta_nominal), (“randomized”, theta_random)): print(f” {name:<11} parameters: “ + “, “.join(f”{k} {v:.3f}” for k, v in decode(th).items())) tasks = np.random.default_rng(1717).integers(0, 10**6, per_cell) print(f”(3) success on {per_cell} held-out tasks per cell”) tables = {} for name, th in ((“tuned on nominal”, theta_nominal), (“tuned with randomization”, theta_random)): tables[name], diverged = grid(pool, th, tasks) print(f” {name}”) show(tables[name]) if diverged.any(): print(“ episodes in which the simulation diverged (counted as failures):”) show(diverged, “”) print(“(4) an open-loop strike”) strikes = part4(pool, per_cell) cells = [(i, j) for i, m in enumerate(GRID[“puck_mass”]) for j, f in enumerate(GRID[“friction”])] print(“mean success inside the randomized range, outside it, and at the nominal values (pusher: grid cells; strike: frictions):”) for name, table in tables.items(): ins = np.mean([table[i, j] for i, j in cells if inside(GRID[“puck_mass”][i], GRID[“friction”][j])]) out = np.mean([table[i, j] for i, j in cells if not inside(GRID[“puck_mass”][i], GRID[“friction”][j])]) print(f” {‘pusher, ‘ + name:<36} inside {ins:.2f}, outside {out:.2f}, nominal {table[2, 2]:.2f}”) inside_mu = [i for i, m in enumerate(MU_GRID) if TRAIN_RANGE[“friction”][0] <= m <= TRAIN_RANGE[“friction”][1]] for name, row in strikes.items(): ins = np.mean([row[i] for i in inside_mu]) out = np.mean([row[i] for i in range(len(MU_GRID)) if i not in inside_mu]) label = “probe, then strike” if name == “probe” else f”strike, mu_hat from {name}” print(f” {label:<36} inside {ins:.2f}, outside {out:.2f}, nominal {row[MU_GRID.index(0.5)]:.2f}”)
Output:
```text
CEM: 24 candidates x 8 episodes per iteration, 12 iterations; randomized range: mass (0.1, 0.4) kg, friction (0.25, 1.0)
(0) puck mass edited in the model, 10 episodes each: diverged episodes
0.2 kg: without mj_setConst 0/10, with it 0/10
0.4 kg: without mj_setConst 10/10, with it 0/10
0.8 kg: without mj_setConst 10/10, with it 0/10
0.4 kg, without mj_setConst but with one derived field recomputed: body_invweight0 10/10, dof_invweight0 10/10, dof_M0 8/10, body_subtreemass 0/10
(1) tuned on nominal dynamics
iteration 1: mean success 0.38, best 0.88
iteration 12: mean success 0.99, best 1.00
(2) tuned with randomized mass and friction
iteration 1: mean success 0.36, best 0.88
iteration 12: mean success 0.88, best 0.88
nominal parameters: speed 0.107, slow_gain 2.168, depth 0.004, lateral 0.011
randomized parameters: speed 0.181, slow_gain 3.256, depth 0.004, lateral 0.010
(3) success on 24 held-out tasks per cell
tuned on nominal
mass \ friction 0.1 0.25 0.5 1.0 1.5
0.05 kg 1.00 1.00 1.00 1.00 1.00
0.10 kg 1.00 1.00 1.00 0.96 1.00
0.20 kg 1.00 1.00 1.00 1.00 0.96
0.40 kg 1.00 1.00 1.00 1.00 1.00
0.80 kg 1.00 1.00 1.00 0.96 1.00
tuned with randomization
mass \ friction 0.1 0.25 0.5 1.0 1.5
0.05 kg 1.00 1.00 1.00 1.00 1.00
0.10 kg 1.00 1.00 1.00 1.00 1.00
0.20 kg 1.00 1.00 1.00 1.00 1.00
0.40 kg 1.00 1.00 1.00 0.96 1.00
0.80 kg 1.00 1.00 1.00 1.00 1.00
(4) an open-loop strike
sliding distance after a launch at 0.4 m/s: simulated against v^2 / (2 mu g)
mu 0.1 simulated 81.2 mm, predicted 81.5 mm
mu 0.25 simulated 32.4 mm, predicted 32.6 mm
mu 0.5 simulated 16.1 mm, predicted 16.3 mm
mu 1.0 simulated 8.2 mm, predicted 8.2 mm
friction estimated by a probe launch (one task), at two probe speeds:
0.15 m/s 0.05: 22.8 mm -> 0.05; 0.25: 4.5 mm -> 0.25; 0.5: 2.5 mm -> 0.46; 1.0: 1.5 mm -> 0.77; 1.5: 1.4 mm -> 0.80
0.3 m/s 0.05: 91.5 mm -> 0.05; 0.25: 18.1 mm -> 0.25; 0.5: 9.2 mm -> 0.50; 1.0: 4.7 mm -> 0.97; 1.5: 3.2 mm -> 1.45
chosen on nominal friction: mu_hat 0.45 (training success 1.00)
chosen on randomized friction: mu_hat 0.7 (training success 0.44)
success against the true friction, 24 held-out tasks each:
true friction 0.05 0.1 0.25 0.5 0.75 1.0 1.25 1.5
mu_hat 0.45 (chosen on nominal) 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00
mu_hat 0.7 (chosen on randomized) 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00
probe at 0.3 m/s, then strike 1.00 1.00 1.00 1.00 1.00 1.00 0.96 0.71
oracle: mu_hat = true mu 1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.54
its largest tilt (deg, median) 0.1 0.2 0.2 0.6 1.3 1.1 3.9 116.1
mean success inside the randomized range, outside it, and at the nominal values (pusher: grid cells; strike: frictions):
pusher, tuned on nominal inside 1.00, outside 0.99, nominal 1.00
pusher, tuned with randomization inside 1.00, outside 1.00, nominal 1.00
strike, mu_hat from nominal inside 0.25, outside 0.00, nominal 1.00
strike, mu_hat from randomized inside 0.25, outside 0.00, nominal 0.00
probe, then strike inside 1.00, outside 0.92, nominal 1.00
Randomized mass makes the simulation diverge, or makes episodes restart. Derived constants are stale. Call mj_setConst after editing mass or inertia, then reset; check data.warning or the clock for silent resets.
Randomized training scores lower than nominal training, and held-out performance is the same. The outcome may not depend on what is randomized. Evaluate the nominal controller across the held-out values before concluding anything.
The randomized controller is best on average and fails at every individual value. It has no information about the parameter; a single action is a compromise. Add an observation that carries the parameter, or identify it.
A measurement of a parameter works in one range and not another. Check its operating range on simulated data where the truth is known, as the probe table does.
A rest check ends a rollout early and changes the results. In the lab, stopping a slide when the puck’s $x$ velocity first fell below $10^{-6}$ m/s changed the probe’s success at friction 1.5 from 0.45 to 0.90: a puck whose sliding velocity has reached zero can still be rocking. Test every degree of freedom.
| For the randomized strike ($\hat\mu$ = 0.7), use the window $ | \hat\mu/\mu - 1 | \le r/D$ and the goal distances PushEnv draws to predict its success at true frictions 0.6, 0.65, …, 0.9. Then measure them with strike_success on the same 24 tasks and compare. |
Replace the free probe launch with information the arm can gather: push the puck for 0.2 s at a fixed speed, release, and record its deceleration. Tune a strike whose $\hat\mu$ is a function of that record (one parameter suffices if you derive it), with friction randomized over 0.25 to 1.0, and evaluate it on the held-out frictions against the probe and the single-value strikes. Report which part of the gain comes from randomization and which from the observation.
[!research] Randomization and adaptation Tobin et al. (arXiv:1703.06907) randomized rendering and trained an object detector only on simulated images with non-realistic random textures; on real images it located objects to 1.5 cm. Peng et al. (arXiv:1710.06537) randomized dynamics for an object-pushing task with a robot arm and report policies “capable of adapting to very different dynamics”; read their design-decision study for what the policy must observe to adapt, the question this lesson’s strike raises. Kumar et al.’s RMA (arXiv:2107.04034) makes the adaptation explicit, with a base policy and an adaptation module, both trained in simulation. When reading such results, check whether the evaluation reaches beyond the training range and whether a nominal-tuned baseline was evaluated on the same values (inference from this lesson): without those two, a randomization gain cannot be told apart from a controller that was insensitive all along. The reading list’s sim-to-real section has more.
{"id": "17.1-check", "title": "Knowledge check", "questions": [
{"kind": "mcq", "q": "A randomized puck mass makes every episode at 0.4 kg diverge. What is the likeliest cause?",
"options": ["The mass range is too wide", "Derived constants were not recomputed with mj_setConst after editing body_mass", "The timestep is too large for heavy objects", "The friction is too high"],
"answer": 1,
"explain": "<p>Output (0): 10/10 diverged without mj_setConst and 0/10 with it; recomputing body_subtreemass alone was enough.</p>"},
{"kind": "mcq", "q": "A pusher tuned on nominal dynamics has a mean success of 0.99 on held-out cells outside the randomized range. What does randomized tuning buy it?",
"options": ["Nothing measurable: the outcome is insensitive to these parameters under this controller", "A large gain outside the range", "A large gain inside the range", "It always helps"],
"answer": 0,
"explain": "<p>Feedback plus a 40 N force margin (against at most 11.8 N needed) leaves nothing for randomization to fix.</p>"},
{"kind": "numeric", "q": "A strike aims for a goal 15 cm away with a 3 cm success radius. By what fraction may mu_hat / mu differ from 1?",
"answer": 0.2, "tol": 0.01,
"explain": "<p>$r/D$ = 0.03/0.15 = 0.2: about ±20%, which is why each single mu_hat succeeds at one grid friction.</p>"},
{"kind": "mcq", "q": "A probe at 0.15 m/s estimates friction 0.80 when the truth is 1.5. What fixes it?",
"options": ["Averaging more probes at 0.15 m/s", "A faster probe, so the slide is long compared with the extra distance the simulation adds near rest", "A smaller timestep", "Randomizing the probe speed"],
"answer": 1,
"explain": "<p>At 0.3 m/s the slide quadruples and the estimate is 1.45.</p>"}
]}
Level 18 turns the probe into a method: estimating physical parameters from measured motion, first by regression and then by fitting simulations.