An environment makes three decisions for whoever learns in it: what an action commands, what is rewarded, and what is observed. Each decision is a hypothesis about the task, and each can be tested before any learning starts. Random actions show where an action space sends the robot. An optimizer, given a reward, shows what that reward actually asks for. Perturbing what no sensor can measure shows whether an observation reads the simulator instead of the world. This lesson runs those three tests on the push task of Lesson 12.1.
[!established] What MuJoCo does and does not constrain A Gymnasium observation is whatever
_get_obsreturns; MuJoCo puts every simulated quantity in reach, from contact forces (mj_contactForce) to model parameters (model.geom_friction). Nothing in either library separates what a robot could sense from what only the simulator knows. That separation has to be declared and tested by the environment’s author.
This lesson has no lab; its pictures are three tables. The one to remember is output (2): the reward that rewards contact with the puck got the highest returns of all four rewards and never solved the task.
Output (1) runs 50 episodes of 100 uniformly random actions (5 s) in each of the three action spaces of ArmCommand:
| Action space | Tool path | Touched the puck | Steps near a joint limit | Arm hit the floor | Steps/s |
|---|---|---|---|---|---|
| torque | 16.0 m | 6/50 | 74% | 50/50 | 1 324 |
| joint targets | 0.85 m | 13/50 | 0% | 37/50 | 1 168 |
| end-effector deltas | 0.84 m | 20/50 | 0% | 0/50 | 355 |
Random torques up to the actuator limits throw the arm around: 16 m of tool travel in 5 s, joints pinned near their limits for three quarters of the time, the arm on the floor in every episode, and the puck knocked 267 mm on average by whichever part of the arm hit it. That is not exploration a learner can use; the policy must first learn gravity compensation before it can learn the task. Joint-target deltas move the tool a plausible distance but know nothing about the floor, which the arm hits in 37 of 50 episodes. End-effector deltas move the tool the same distance in the plane where the task lives, at a height the controller holds: the floor is never hit and the puck is touched most often.
The cost is in two places. The end-effector controller is the slowest (355 environment steps per second against 1 324; rates vary between runs with the machine’s load), and it builds task knowledge into the environment: the tool cannot leave its height or tilt, so a policy can never discover a strategy that needs either. Choosing an action space is choosing which part of the problem the learner solves and which part the engineer has already solved.
Output (2) gives the same task (puck at (0.47, 0), goal at (0.62, 0.10)) four rewards and optimizes each one with the cross-entropy method: a plan of six piecewise-constant end-effector actions over 60 steps (12 numbers), 32 plans per iteration, the best fifth refitted, 20 iterations, 5 seeds. The rewards per step:
\[r_\text{dense} = -\|p - g\|,\qquad r_\text{shaped} = r_\text{dense} - \tfrac12\|x_\text{tool} - p\|,\] \[r_\text{sparse} = \mathbb{1}[\|p - g\| < 3\text{ cm}],\qquad r_\text{contact} = \mathbb{1}[\text{tool touches puck}],\]with $p$ the puck’s position and $g$ the goal. Success, in every case, is the puck within 3 cm of the goal.
| Reward | Task success | Final puck-goal distance |
|---|---|---|
| dense | 3/5 | 15 to 81 mm |
| shaped | 4/5 | 20 to 65 mm |
| sparse | 0/5 | 129 to 180 mm |
| contact | 0/5 | 131 to 199 mm |
Two failures, two different reasons. The sparse reward is zero for every plan the optimizer samples, so the elite set is a random choice and the search never leaves its start (the puck ends where it began, 180 mm from the goal, in three of five seeds). That is an exploration failure: the reward is correct and uninformative. The contact reward is informative and wrong: the optimizer keeps the tool against the puck for 34 to 46 of the 60 steps, the highest returns in the table, and the puck ends 131 to 199 mm from the goal, against 180 mm at the start. That is reward hacking: the optimizer did exactly what was rewarded, and the direction the puck travels is not part of it.
The dense and shaped rewards succeed in 7 of 10 runs together, against 0 of 10 for the other two (Fisher’s exact test, p = 0.003). Whether shaping helps is not settled here: 4 of 5 against 3 of 5 gives p = 1.0. In the dense reward’s two failures the puck was pushed partway (76 and 81 mm from the goal) and the search settled there, a local optimum of the plan parameters; five seeds cannot say whether the shaping term makes that less likely.
[!recommendation] Optimize a reward before you train on it A cheap optimizer (random search, cross-entropy over a few plan parameters, a scripted policy family) run against a new reward shows whether it is satisfiable, whether it is informative, and what behaviour maximizes it. Watch the highest-return behaviour, not the return: a reward that is maximized without solving the task will be maximized by your learner too, only more slowly and less visibly.
The push observation is declared component by component with the real sensor that would provide each one: joint encoders, their differentiation, forward kinematics, a camera tracker for the puck, and the task specification for the goal (OBSERVATION in push.py). Output (3) runs two tests from mjcourse.envs.checks on the standard observation and on four leaky variants:
undeclared_sources fails any component whose declared source is not a real sensor. It catches the true puck velocity (simulator state), the contact force (contact solver) and the puck’s friction coefficient (model parameter).perturbation_leaks changes, at one instant, quantities no sensor reports (the object’s velocity, mass, friction, and contact stiffness), recomputes the observation and fails if it moved. It catches the same three, and it catches the fourth, which the first test passes: the puck’s true velocity declared as coming from the camera tracker.The second test is the stronger one because it checks behaviour rather than labels. Its limit is its list: a leak through a quantity not in the list (say, the solver’s iteration count) goes unnoticed. The contact-force leak shows how far a leak reaches: it changes with the puck’s velocity, mass and contact stiffness, because the solver’s force depends on all three.
INPUT: mjcourse.envs.PushEnv and mjcourse.envs.checks
PROCESS: random rollouts per action space; the cross-entropy method on four rewards; source and perturbation tests on five observations
OUTPUT: printed tables
The push task and the checks:
```python file=src/mjcourse/envs/push.py “"”Push: slide a puck to a goal with the closed gripper (Level 12 checkpoint).
Observation (float32, 21): joint positions (7), joint velocities (7), tcp position (3),
puck position xy (2), goal xy (2). Every component has a declared real-robot source in
OBSERVATION; Lesson 12.2’s test fails the environment if one does not.
extra_observations appends components a real robot could not measure, for that test
to catch: “puck_velocity” (simulator state), “contact_force” (contact solver),
“puck_friction” (model parameter).
Action: see mjcourse.envs.arm; “ee_delta” moves the tcp in x and y at a fixed height.
Reward (reward=): “dense” minus the puck-goal distance; “shaped” adds minus half the
tcp-puck distance; “sparse” 1 on success; “contact” 1 per step in contact with the puck.
Terminated: the puck is within success_radius of the goal, unless terminate_on_success
is False (then success is judged where the puck ends). Truncated: after
max_episode_steps steps (0.05 s each by default).
reset(options=…) accepts “puck” and “goal” positions and, for Level 17, “puck_mass” (kg)
and “friction” (sliding coefficient of the puck and the floor); parameters not given
return to their nominal values, since model edits persist across episodes.
“””
from future import annotations
import mujoco import numpy as np from gymnasium import spaces
from mjcourse import kinematics, spatial from mjcourse.envs.arm import ArmCommand from mjcourse.envs.base import MujocoEnv
OBSERVATION = ((“joint_pos”, 7, “joint encoders”), (“joint_vel”, 7, “joint encoders, differentiated”), (“tcp_pos”, 3, “forward kinematics of the joint encoders”), (“puck_xy”, 2, “camera tracker”), (“goal_xy”, 2, “task specification”)) PRIVILEGED = {“puck_velocity”: (3, “simulator state”), “contact_force”: (1, “contact solver”), “puck_friction”: (1, “model parameter”)} REWARDS = (“dense”, “shaped”, “sparse”, “contact”) PUSH_HEIGHT = 0.03 # tcp height while pushing (m) START_TCP = np.array([0.36, 0.0, PUSH_HEIGHT])
class PushEnv(MujocoEnv): _start_q: np.ndarray | None = None # shared IK solution for the start pose
def __init__(self, action_mode: str = "ee_delta", reward: str = "shaped", frame_skip: int = 25,
max_episode_steps: int = 100, success_radius: float = 0.03, extra_observations: tuple[str, ...] = (),
terminate_on_success: bool = True, max_lead: float | None = None, **kwargs):
super().__init__("push", frame_skip, max_episode_steps, **kwargs)
if reward not in REWARDS:
raise ValueError(f"reward must be one of {REWARDS}")
self.reward_kind, self.success_radius, self.extra = reward, success_radius, tuple(extra_observations)
self.terminate_on_success = terminate_on_success
self.arm = ArmCommand(self.model, self.data, action_mode, "gripper/tcp", ee_axes=(0, 1),
ee_low=(0.25, -0.35, PUSH_HEIGHT), ee_high=(0.75, 0.35, PUSH_HEIGHT), max_lead=max_lead)
self.action_space = self.arm.space
self.observation_sources = {name: source for name, _, source in OBSERVATION}
self.observation_sources |= {name: PRIVILEGED[name][1] for name in self.extra}
size = sum(n for _, n, _ in OBSERVATION) + sum(PRIVILEGED[name][0] for name in self.extra)
self.observation_space = spaces.Box(-np.inf, np.inf, shape=(size,), dtype=np.float32)
self.puck = self.model.body("puck").id
self.puck_geom = self.model.geom("puck").id
self.goal_site = self.model.site("goal").id
self.pusher = {self.model.geom(n).id for n in ("gripper/pad_left", "gripper/pad_right",
"gripper/finger_left", "gripper/finger_right")}
self.puck_adr = self.model.jnt_qposadr[self.model.joint("puck").id]
self.floor_geom = self.model.geom("floor").id
self.nominal = {"puck_mass": float(self.model.body_mass[self.puck]),
"puck_inertia": self.model.body_inertia[self.puck].copy(),
"friction": float(self.model.geom_friction[self.puck_geom, 0])}
if PushEnv._start_q is None:
mujoco.mj_resetDataKeyframe(self.model, self.data, self.keyframe)
mujoco.mj_forward(self.model, self.data)
down = spatial.mat_to_quat(self.data.site("gripper/tcp").xmat.reshape(3, 3))
result, _ = kinematics.solve_ik_restarts(self.model, "gripper/tcp", START_TCP, down, q_init=self.data.qpos.copy(),
joints=list(range(7)), restarts=10, seed=0, tol_pos=5e-4, tol_rot=5e-3)
PushEnv._start_q = result.qpos[:7].copy()
def _reset_task(self, options: dict) -> None:
d, rng, m = self.data, self.np_random, self.model
mass = options.get("puck_mass", self.nominal["puck_mass"])
if mass != m.body_mass[self.puck]:
m.body_mass[self.puck] = mass
m.body_inertia[self.puck] = self.nominal["puck_inertia"] * mass / self.nominal["puck_mass"]
mujoco.mj_setConst(m, d) # derived constants (inverse weights that set contact softness)
mujoco.mj_resetDataKeyframe(m, d, self.keyframe) # mj_setConst used d as scratch
for g in (self.puck_geom, self.floor_geom):
m.geom_friction[g, 0] = options.get("friction", self.nominal["friction"])
d.qpos[:7] = PushEnv._start_q
d.ctrl[7] = 0.0 # gripper closed: a pusher
d.qpos[7:9] = 0.0
puck = options.get("puck", np.array([rng.uniform(0.44, 0.50), rng.uniform(-0.05, 0.05)]))
d.qpos[self.puck_adr:self.puck_adr + 7] = [*puck, 0.02, 1, 0, 0, 0]
goal = options.get("goal", np.array([rng.uniform(0.58, 0.66), rng.uniform(-0.15, 0.15)]))
self.model.site_pos[self.goal_site][:2] = goal
mujoco.mj_forward(self.model, d)
self.arm.reset()
def _apply_action(self, action: np.ndarray) -> None:
self.arm.apply(action)
def _control(self) -> None:
self.arm.control()
self.data.ctrl[7] = 0.0
def puck_xy(self) -> np.ndarray:
return self.data.xpos[self.puck][:2].copy()
def goal_xy(self) -> np.ndarray:
return self.model.site_pos[self.goal_site][:2].copy()
def _distance(self) -> float:
return float(np.linalg.norm(self.puck_xy() - self.goal_xy()))
def _in_contact(self) -> bool:
d = self.data
return any((c.geom1 == self.puck_geom and c.geom2 in self.pusher) or (c.geom2 == self.puck_geom and c.geom1 in self.pusher)
for c in d.contact[:d.ncon])
def _get_obs(self) -> np.ndarray:
d = self.data
parts = [d.qpos[:7], d.qvel[:7], d.site("gripper/tcp").xpos, self.puck_xy(), self.goal_xy()]
for name in self.extra:
if name == "puck_velocity":
velocity = np.zeros(6) # [angular; linear] in the world frame
mujoco.mj_objectVelocity(self.model, d, mujoco.mjtObj.mjOBJ_BODY, self.puck, velocity, 0)
parts.append(velocity[3:])
elif name == "contact_force":
total, f = 0.0, np.zeros(6)
for i in range(d.ncon):
if self.puck_geom in (d.contact[i].geom1, d.contact[i].geom2):
mujoco.mj_contactForce(self.model, d, i, f)
total += f[0]
parts.append([total])
elif name == "puck_friction":
parts.append([self.model.geom_friction[self.puck_geom, 0]])
return np.concatenate(parts).astype(np.float32)
def _reward(self, action: np.ndarray) -> float:
if self.reward_kind == "dense":
return -self._distance()
if self.reward_kind == "shaped":
return -self._distance() - 0.5 * float(np.linalg.norm(self.data.site("gripper/tcp").xpos[:2] - self.puck_xy()))
if self.reward_kind == "sparse":
return float(self._distance() < self.success_radius)
return float(self._in_contact())
def _terminal(self) -> bool:
return self.terminate_on_success and self._distance() < self.success_radius
def _get_info(self) -> dict:
return {"distance": self._distance(), "is_success": self._distance() < self.success_radius,
"contact": self._in_contact()} ``` ```python file=src/mjcourse/envs/checks.py """Checks gymnasium's env_checker does not make (Lessons 12.1 and 12.2).
Each returns a value a test can assert on: observations_are_copies(env) stored observations do not change after later steps truncation_is_reported(env) the time limit gives truncated=True, terminated=False reproducible(env, seed) the same seed and actions give the same episode undeclared_sources(env, allowed) observation components whose declared source is not a real sensor perturbation_leaks(env, …) quantities no sensor reports that nevertheless change the observation “””
from future import annotations
import mujoco import numpy as np
REAL_SOURCES = frozenset({“joint encoders”, “joint encoders, differentiated”, “forward kinematics of the joint encoders”, “camera tracker”, “task specification”})
def observations_are_copies(env, steps: int = 3, seed: int = 0) -> bool: stored = [env.reset(seed=seed)[0]] snapshot = [stored[0].copy()] for _ in range(steps): obs = env.step(env.action_space.sample())[0] stored.append(obs) snapshot.append(obs.copy()) return all(np.array_equal(a, b) for a, b in zip(stored, snapshot))
def truncation_is_reported(env, seed: int = 0) -> bool: “"”Runs a zero action to the time limit; true if the last step is truncated and not terminated (and the episode did not end earlier).””” env.reset(seed=seed) zero = np.zeros(env.action_space.shape, env.action_space.dtype) for t in range(env.max_episode_steps): _, _, terminated, truncated, _ = env.step(zero) last = t == env.max_episode_steps - 1 if terminated or truncated: return last and truncated and not terminated return False
def reproducible(make_env, seed: int = 0, steps: int = 20) -> bool: actions = np.random.default_rng(seed).uniform(-1, 1, (steps,) + make_env().action_space.shape).astype(np.float32) runs = [] for _ in range(2): env = make_env() obs = [env.reset(seed=seed)[0]] obs += [env.step(a)[0] for a in actions] runs.append(np.array(obs)) env.close() return np.array_equal(runs[0], runs[1])
def undeclared_sources(env, allowed=REAL_SOURCES) -> list[str]: return [f”{name} ({source})” for name, source in env.observation_sources.items() if source not in allowed]
def perturbation_leaks(env, body: str = “puck”, geom: str = “puck”, seed: int = 0, warmup: int = 20) -> list[str]: “"”At one instant, change quantities no real sensor reports (the object’s velocity, mass, friction and contact stiffness) and recompute the observation; it must not change.””” env.reset(seed=seed) push = np.zeros(env.action_space.shape, env.action_space.dtype) push[0] = 1.0 for _ in range(warmup): # move, and touch the object env.step(push) m, d = env.model, env.data b, g = m.body(body).id, m.geom(geom).id dof = m.body_dofadr[b] spec = mujoco.mjtState.mjSTATE_FULLPHYSICS mujoco.mj_forward(m, d) # baseline from a fresh state, so a merely stale base = env._get_obs().copy() # observation is not reported as a leak
def bump_velocity():
d.qvel[dof:dof + m.body_dofnum[b]] += 0.3
perturbations = {"object velocity": bump_velocity,
"object mass": lambda: m.body_mass.__setitem__(b, 2 * m.body_mass[b]),
"object friction": lambda: m.geom_friction.__setitem__(g, 2 * m.geom_friction[g]),
"contact stiffness": lambda: m.geom_solref.__setitem__(g, [0.005, 1.0])}
leaks = []
for name, apply in perturbations.items():
state = np.zeros(mujoco.mj_stateSize(m, spec))
mujoco.mj_getState(m, d, state, spec)
saved = m.body_mass.copy(), m.geom_friction.copy(), m.geom_solref.copy()
apply()
mujoco.mj_forward(m, d)
if not np.array_equal(env._get_obs(), base):
leaks.append(name)
m.body_mass[:], m.geom_friction[:], m.geom_solref[:] = saved
mujoco.mj_setState(m, d, state, spec)
mujoco.mj_forward(m, d)
return leaks ``` ```python file=examples/l12_2_env_design.py """Lesson 12.2: action spaces, rewards and observations of the push task, measured.
INPUT mjcourse.envs.PushEnv PROCESS (1) random actions in each action space (torque, joint_delta, ee_delta): how far the tool travels, whether it reaches the puck, joint-limit and floor contacts, cost; (2) four rewards (dense, shaped, sparse, contact) each optimized by the cross-entropy method over open-loop plans: the return found against the task’s success; (3) privileged information, with mjcourse.envs.checks: a test of declared observation sources and a test that perturbs quantities a robot cannot sense, on the standard observation and on four leaky ones (one of them mislabelled) OUTPUT printed tables
Run: python examples/l12_2_env_design.py “””
import os import time
import gymnasium as gym import mujoco import numpy as np
import mjcourse.envs # noqa: F401 (registers the environments) from mjcourse.envs import checks from mjcourse.envs.push import OBSERVATION, PushEnv
FAST = os.environ.get(“MJC_FAST”) == “1” TASK = {“puck”: np.array([0.47, 0.0]), “goal”: np.array([0.62, 0.10])} # one fixed task for part (2) HORIZON, KNOTS = 60, 6 # 3 s plans with 6 piecewise-constant knots
def random_rollouts(mode: str, episodes: int) -> dict: env = PushEnv(action_mode=mode, reward=”dense”) m, d = env.model, env.data floor = m.geom(“floor”).id arm_bodies = {b for b in range(m.nbody) if b not in (0, m.body(“puck”).id)} out = {“path”: [], “reach”: 0, “moved”: [], “near_limit”: [], “floor”: 0} start = time.perf_counter() steps = 0 for i in range(episodes): env.reset(seed=i) env.action_space.seed(i) tcp_prev, path, touched, floor_hit, limit_steps = d.site(“gripper/tcp”).xpos.copy(), 0.0, False, False, 0 puck0 = env.puck_xy() for _ in range(env.max_episode_steps): _, _, terminated, _, info = env.step(env.action_space.sample()) steps += 1 tcp = d.site(“gripper/tcp”).xpos.copy() path += np.linalg.norm(tcp - tcp_prev) tcp_prev = tcp touched |= info[“contact”] q, lo, hi = d.qpos[:7], m.jnt_range[:7, 0], m.jnt_range[:7, 1] limit_steps += bool(np.any((q - lo < 0.02 * (hi - lo)) | (hi - q < 0.02 * (hi - lo)))) floor_hit |= any((c.geom1 == floor and m.geom_bodyid[c.geom2] in arm_bodies) or (c.geom2 == floor and m.geom_bodyid[c.geom1] in arm_bodies) for c in d.contact[:d.ncon]) if terminated: break out[“path”].append(path) out[“reach”] += touched out[“moved”].append(np.linalg.norm(env.puck_xy() - puck0)) out[“near_limit”].append(limit_steps / env.max_episode_steps) out[“floor”] += floor_hit out[“rate”] = steps / (time.perf_counter() - start) return out
def part1() -> None: n = 10 if FAST else 50 print(f” {n} episodes of 100 random actions (5 s) per mode”) print(f” {‘mode’:<12}{‘tool path (m)’:>14}{‘touched puck’:>14}{‘puck moved (mm)’:>17}{‘steps near a limit’:>20}{‘arm hit floor’:>15}{‘steps/s’:>9}”) for mode in (“torque”, “joint_delta”, “ee_delta”): r = random_rollouts(mode, n) print(f” {mode:<12}{np.mean(r[‘path’]):>14.2f}{r[‘reach’]:>11d}/{n}{1000 * np.mean(r[‘moved’]):>17.1f}” f”{np.mean(r[‘near_limit’]):>20.1%}{r[‘floor’]:>12d}/{n}{r[‘rate’]:>9.0f}”)
def plan_actions(params: np.ndarray) -> np.ndarray: “”“(KNOTS*2,) parameters -> (HORIZON, 2) piecewise-constant ee_delta actions.””” return np.repeat(np.clip(params.reshape(KNOTS, 2), -1, 1), HORIZON // KNOTS, axis=0).astype(np.float32)
def evaluate(envs, population: np.ndarray) -> tuple[np.ndarray, np.ndarray, np.ndarray]: “"”Returns, success flags and final puck-goal distances of each plan, run in parallel.””” n = len(population) envs.reset(options=TASK) plans = np.stack([plan_actions(p) for p in population]) returns, success, done = np.zeros(n), np.zeros(n, bool), np.zeros(n, bool) distance = np.zeros(n) for t in range(HORIZON): _, reward, terminated, truncated, info = envs.step(plans[:, t]) live = ~done returns[live] += reward[live] success[live] |= info[“is_success”][live] distance[live] = info[“distance”][live] done |= terminated | truncated if done.all(): break return returns, success, distance
def cem(reward: str, seed: int, population: int, iterations: int, envs) -> dict: rng = np.random.default_rng(seed) mean, std = np.zeros(KNOTS * 2), np.full(KNOTS * 2, 0.6) for _ in range(iterations): samples = np.clip(rng.normal(mean, std, (population, KNOTS * 2)), -1, 1) returns, _, _ = evaluate(envs, samples) elite = samples[np.argsort(returns)[-max(2, population // 5):]] mean, std = elite.mean(axis=0), elite.std(axis=0) + 0.02 returns, success, distance = evaluate(envs, np.repeat(mean[None], population, axis=0)[:population]) return {“return”: returns[0], “success”: bool(success[0]), “distance”: distance[0]}
def part2() -> None: population, iterations, seeds = (8, 3, 2) if FAST else (32, 20, 5) print(f” task: puck at {TASK[‘puck’]}, goal at {TASK[‘goal’]}; plans of {KNOTS} piecewise-constant ee_delta “ f”actions over {HORIZON} steps; CEM population {population}, {iterations} iterations, {seeds} seeds per reward”) print(f” {‘reward optimized’:<18}{‘return of the plan (per seed)’:<40}{‘task success’:>13}{‘final puck-goal distance (mm)’:>36}”) for reward in (“dense”, “shaped”, “sparse”, “contact”): envs = gym.vector.AsyncVectorEnv([lambda r=reward: gym.make(“mjcourse/Push-v0”, action_mode=”ee_delta”, reward=r, max_episode_steps=HORIZON)] * population) results = [cem(reward, s, population, iterations, envs) for s in range(seeds)] envs.close() print(f” {reward:<18}{‘, ‘.join(format(r[‘return’], ‘.2f’) for r in results):<40}” f”{sum(r[‘success’] for r in results):>10d}/{seeds}” f”{‘, ‘.join(format(1000 * r[‘distance’], ‘.0f’) for r in results):>36}”)
class MislabelledLeak(PushEnv): “"”Appends the puck’s true velocity but declares it as coming from the camera tracker.”””
def __init__(self, **kwargs):
super().__init__(extra_observations=("puck_velocity",), **kwargs)
self.observation_sources["puck_velocity"] = "camera tracker"
def part3() -> None: print(f” declared sources allowed: {sorted(checks.REAL_SOURCES)}”) print(f” {‘observation’:<44}{‘declared-source test’:<42}{‘perturbation test (changed by)’:<30}”) variants = [(“standard: “ + “, “.join(n for n, _, _ in OBSERVATION), PushEnv(action_mode=”ee_delta”)), (“+ puck velocity (simulator state)”, PushEnv(action_mode=”ee_delta”, extra_observations=(“puck_velocity”,))), (“+ contact force (contact solver)”, PushEnv(action_mode=”ee_delta”, extra_observations=(“contact_force”,))), (“+ puck friction (model parameter)”, PushEnv(action_mode=”ee_delta”, extra_observations=(“puck_friction”,))), (“+ puck velocity labelled ‘camera tracker’”, MislabelledLeak(action_mode=”ee_delta”))] for label, env in variants: bad, leaks = checks.undeclared_sources(env), checks.perturbation_leaks(env) print(f” {label[:43]:<44}{‘pass’ if not bad else ‘FAIL: ‘ + ‘, ‘.join(bad):<42}{‘pass’ if not leaks else ‘FAIL: ‘ + ‘, ‘.join(leaks):<30}”) env.close()
if name == “main”: print(f”MuJoCo {mujoco.version}, gymnasium {gym.version}”) print(“(1) action spaces under random actions”) part1() print(“(2) rewards optimized by the cross-entropy method”) part2() print(“(3) privileged information in observations”) part3()
Output:
```text
MuJoCo 3.14.0, gymnasium 1.3.0
(1) action spaces under random actions
50 episodes of 100 random actions (5 s) per mode
mode tool path (m) touched puck puck moved (mm) steps near a limit arm hit floor steps/s
torque 15.99 6/50 266.8 74.3% 50/50 1324
joint_delta 0.85 13/50 12.6 0.0% 37/50 1168
ee_delta 0.84 20/50 16.3 0.0% 0/50 355
(2) rewards optimized by the cross-entropy method
task: puck at [0.47 0. ], goal at [0.62 0.1 ]; plans of 6 piecewise-constant ee_delta actions over 60 steps; CEM population 32, 20 iterations, 5 seeds per reward
reward optimized return of the plan (per seed) task success final puck-goal distance (mm)
dense -5.33, -2.92, -5.59, -1.94, -1.87 3/5 76, 29, 81, 15, 16
shaped -2.69, -3.66, -8.30, -3.02, -4.14 4/5 28, 28, 65, 20, 28
sparse 0.00, 0.00, 0.00, 0.00, 0.00 0/5 180, 180, 129, 180, 133
contact 46.00, 36.00, 34.00, 38.00, 46.00 0/5 172, 152, 131, 199, 161
(3) privileged information in observations
declared sources allowed: ['camera tracker', 'forward kinematics of the joint encoders', 'joint encoders', 'joint encoders, differentiated', 'task specification']
observation declared-source test perturbation test (changed by)
standard: joint_pos, joint_vel, tcp_pos, pu pass pass
+ puck velocity (simulator state) FAIL: puck_velocity (simulator state) FAIL: object velocity
+ contact force (contact solver) FAIL: contact_force (contact solver) FAIL: object velocity, object mass, contact stiffness
+ puck friction (model parameter) FAIL: puck_friction (model parameter) FAIL: object friction
+ puck velocity labelled 'camera tracker' pass FAIL: object velocity
The course’s test suite (tests/test_envs.py) runs check_env, the three checks of Lesson 12.1 and both leakage tests on every environment and variant; it is the reference answer to the checkpoint’s last requirement.
A policy learns quickly in simulation and is useless on the robot. Look for privileged observations first: contact flags, true object velocities, model parameters. Run the perturbation test.
Returns rise and the task is not solved. Reward hacking. Roll out the highest-return behaviour and watch it; then remove the loophole or reward the outcome instead of a proxy.
Returns stay flat from the first iteration. A sparse reward that no sampled behaviour ever reaches. Shape it, start episodes closer to success, or add demonstrations (Level 14).
Random exploration never touches the object. The action space spends its variance where the task is not (torques fighting gravity, joints moving the arm into the floor). Move the controller’s knowledge into the action space, knowing what it rules out.
Add a fifth reward, the dense reward plus a bonus of 1 on success, and run part (2) with 10 seeds for it and for the dense reward. Report both success rates with Wilson intervals and say whether the bonus made a difference you can detect.
Train the same task with each action space using the PPO of Lesson 13.2 (5 seeds, 300 000 environment steps each) and compare final success rates and wall-clock time. Predict the ranking from output (1) before you run it, and explain any surprise.
[!research] Privileged information, used on purpose Privileged information is not always a mistake. Asymmetric actor-critic methods give the critic full simulator state while the actor sees only what the robot will see (Pinto et al., arXiv:1710.06542), and teacher-student schemes train a privileged teacher and distil it into a student with realistic inputs. Both keep the leak on the training side of the line, which is what the tests of this lesson enforce for the policy’s input. The design question for a research environment is then not whether to use privileged information but where, and the perturbation test gives a way to state and check the answer.
{"id": "12.2-check", "title": "Knowledge check", "questions": [
{"kind": "mcq", "q": "Random torques moved the tool 16 m in 5 s and put the arm on the floor in every episode. What does that say about torque actions for learning this task?",
"options": ["Torque actions explore best", "The learner must first learn to compensate gravity before random variation becomes useful exploration", "Torques are not allowed in Gymnasium", "The simulation is unstable"],
"answer": 1,
"explain": "<p>Output (1): the variance goes into fighting gravity and joint limits, not into reaching the puck.</p>"},
{"kind": "mcq", "q": "A reward is optimized to the highest returns seen, and the task is never solved. What is this called, and what do you do first?",
"options": ["An exploration failure; make the reward denser", "Reward hacking; roll out the highest-return behaviour and look at it", "Overfitting; add more seeds", "A sparse reward; add a bonus"],
"answer": 1,
"explain": "<p>Output (2), contact reward: 34 to 46 steps of contact out of 60 and 0/5 successes.</p>"},
{"kind": "mcq", "q": "An observation component is declared as coming from a camera tracker but is the simulator's true velocity. Which test catches it?",
"options": ["The declared-source test", "The perturbation test", "gymnasium's check_env", "None of them"],
"answer": 1,
"explain": "<p>Changing the puck's velocity at one instant changes the observation, whatever the label says (output 3, last row).</p>"},
{"kind": "numeric", "q": "Dense and shaped rewards succeeded in 7 of 10 runs, sparse and contact in 0 of 10. Using Fisher's exact test, what is the two-sided p-value (to two significant figures)?",
"answer": 0.0031, "tol": 0.0002,
"explain": "<p>p = 0.0031. The comparison of dense (3/5) with shaped (4/5) gives p = 1.0: not separated by five seeds.</p>"}
]}
Level 13 learns in these environments: first a policy search you can watch, then PPO written from scratch, then how to evaluate any agent so that its numbers mean something.