Core concept

Gymnasium fixes the contract between a learning algorithm and a task: reset(seed=...) returns an observation and an info dictionary; step(action) returns the next observation, a reward, whether the task ended (terminated), whether time ran out (truncated), and info. On MuJoCo, every part of that contract is a decision about MjModel and MjData: which keyframe an episode starts from, what randomness the task draws and from which generator, how many physics steps one action is held for, which quantities the observation reads and when, and what ends an episode.

This lesson writes that contract out with no library base class in between (mjcourse.envs.base.MujocoEnv), builds two tasks on it (reach and push, each with three ways to command the arm), and then tests it. Gymnasium’s own checker passes everything correct here, but it also passes two of four deliberately broken versions, so the lesson measures what those two cost.

[!established] The Gymnasium contract terminated means the episode reached a terminal state of the task (success, failure); truncated means it was cut off for a reason outside the task, usually a time limit. Learning algorithms treat them differently: after termination the future value is zero, after truncation it is not. reset(seed=s) seeds the environment’s generator self.np_random; reset() without a seed continues it. (Gymnasium documentation, Env.step and Env.reset.)

Visual intuition

This lesson has no lab: an environment is a contract, and the lesson’s figures are the tables its script prints. Read output (6) as the picture. A random policy on the reach task almost never ends the episode by itself (98 of 100 episodes are truncated at 100 steps); a policy that moves toward the target ends every episode by success, after 10.7 steps on average. The same environment, with the same reward, produces two very different mixtures of terminal and truncated transitions, and a learning algorithm must handle both correctly.

Mathematics

One environment step

An environment step of length $\Delta t$ holds the action for frame_skip physics steps of length $h$, $\Delta t = n_\text{skip} h$; here $h$ = 2 ms and $n_\text{skip}$ = 25, so the policy acts at 20 Hz while the physics and the inner controller run at 500 Hz. Inside the step, _apply_action turns the action into a controller target once, _control runs the controller before every physics step, and the observation is read after a final mj_forward.

That last call matters. mj_step computes positions, contacts and forces at the start of the step and then integrates, so after it returns, qpos is new and site_xpos is one step old. Output (4) measures the inconsistency under random torques: the end effector read without mj_forward is 1.77 mm away on average (5.9 mm at worst) from where qpos puts it, which is the end effector’s speed times one timestep. An observation built from both qpos and site positions mixes two instants unless mj_forward runs first.

What the checker does and does not catch

Output (2) runs gymnasium.utils.env_checker.check_env on the six variants (all pass) and on four broken ones:

The two it passes are the expensive ones. A view is a window onto MjData’s memory: output (3) stores four observations from a view, and they compare equal, because all four are the current state. A replay buffer filled this way holds one state many times, and nothing raises an error. Reporting the time limit as termination is invisible to any check of types, because it is a statement about meaning: it tells the learner that the states where time ran out are terminal, so their value is zero. With a reward of minus the distance per step, a state where the arm is still 0.3 m away and the clock stops looks like a state with no future cost (Level 13 measures the effect on learning).

[!recommendation] Test what the checker cannot Add three tests of your own to every environment: stored observations do not change after later steps (copy, never return views of MjData); an episode cut by the time limit returns truncated=True, terminated=False; and the same seed and actions reproduce the episode exactly. All three are in this lesson’s script; the course’s test suite runs them on both environments.

Seeding

reset(seed=3) twice with the same 50 actions reproduces the episode exactly, and seed 4 does not (output 5). Calling reset(seed=7) once and then reset() three times gives four different targets, and two fresh environments called the same way give the same four. That is the behaviour learning code expects: seed once per environment, then let the stream continue. Re-seeding every reset with the same seed would make every episode identical.

Throughput

Output (7) times one process. Raw torques give 5 831 environment steps per second on the reach task (146 000 physics steps per second); joint targets with a PD loop give 2 919; end-effector targets with the Cartesian impedance of Lesson 10.3 give 465, because that controller runs in Python at 500 Hz and computes the mass matrix and a null-space projection every physics step. The controller inside the environment, not the physics, sets the cost. Vector environments in separate processes raise the total to about 12 000 steps per second at 16 processes, and 32 processes add almost nothing: past that point the parent process, which sends actions and gathers observations, is the bottleneck (inference from the plateau; the script does not profile it). Absolute rates vary with the machine’s load from run to run.

Implementation

INPUT: mjcourse.envs (MujocoEnv, ArmCommand, ReachEnv, PushEnv)
PROCESS: check the environments, break them on purpose, measure stale kinematics, seeding, termination statistics and throughput
OUTPUT: printed tables

The base class, the reach task and the action modes:

```python file=src/mjcourse/envs/base.py “"”A Gymnasium environment written directly on MuJoCo (Lesson 12.1).

Nothing is hidden behind a library base class: the model is loaded, the episode is reset from a keyframe, the task randomizes its initial state from the environment’s own generator (self.np_random, seeded by reset(seed=...)), each step holds one action for frame_skip physics steps, and the episode is truncated by the environment itself after max_episode_steps.

Subclasses implement _reset_task, _apply_action, _get_obs, _reward and _terminal, and declare observation_space and action_space. “””

from future import annotations

import gymnasium as gym import mujoco import numpy as np

from mjcourse.paths import model_path

class MujocoEnv(gym.Env): metadata = {“render_modes”: [“rgb_array”], “render_fps”: 20}

def __init__(self, model: str, frame_skip: int, max_episode_steps: int, keyframe: str = "home",
             render_mode: str | None = None, camera: str | int = -1, width: int = 320, height: int = 240):
    self.model = mujoco.MjModel.from_xml_path(str(model_path(model)))
    self.data = mujoco.MjData(self.model)
    self.frame_skip, self.max_episode_steps = frame_skip, max_episode_steps
    self.dt = self.model.opt.timestep * frame_skip          # seconds per environment step
    self.metadata = {**self.metadata, "render_fps": round(1.0 / self.dt)}
    self.keyframe = self.model.key(keyframe).id
    if render_mode is not None and render_mode not in self.metadata["render_modes"]:
        raise ValueError(f"render_mode must be one of {self.metadata['render_modes']}")
    self.render_mode, self.camera, self.size = render_mode, camera, (width, height)
    self._renderer: mujoco.Renderer | None = None
    self._steps = 0

# ------------------------------------------------------------- Gymnasium API

def reset(self, *, seed: int | None = None, options: dict | None = None):
    super().reset(seed=seed)                                  # seeds self.np_random on the first call or when given
    mujoco.mj_resetDataKeyframe(self.model, self.data, self.keyframe)
    self._reset_task(options or {})
    mujoco.mj_forward(self.model, self.data)
    self._steps = 0
    return self._get_obs(), self._get_info()

def step(self, action):
    action = np.asarray(action, dtype=np.float64)
    if action.shape != self.action_space.shape:
        raise ValueError(f"action shape {action.shape}, expected {self.action_space.shape}")
    action = np.clip(action, self.action_space.low, self.action_space.high)
    self._apply_action(action)                                # sets controller targets or ctrl
    for _ in range(self.frame_skip):
        self._control()                                       # inner controller at the physics rate
        mujoco.mj_step(self.model, self.data)
    mujoco.mj_forward(self.model, self.data)                  # positions after the last step, not before it
    self._steps += 1
    obs = self._get_obs()
    terminated = self._terminal()                             # the task decided: success or failure
    truncated = self._steps >= self.max_episode_steps and not terminated   # time ran out
    return obs, float(self._reward(action)), terminated, truncated, self._get_info()

def render(self):
    if self.render_mode != "rgb_array":
        return None
    if self._renderer is None:
        self._renderer = mujoco.Renderer(self.model, self.size[1], self.size[0])
    self._renderer.update_scene(self.data, self.camera)
    return self._renderer.render()

def close(self):
    if self._renderer is not None:
        self._renderer.close()
        self._renderer = None

# ------------------------------------------------------------- for subclasses

def _reset_task(self, options: dict) -> None:
    raise NotImplementedError

def _apply_action(self, action: np.ndarray) -> None:
    raise NotImplementedError

def _control(self) -> None:
    """Called before every physics step; the default leaves data.ctrl as it is."""

def _get_obs(self) -> np.ndarray:
    raise NotImplementedError

def _reward(self, action: np.ndarray) -> float:
    raise NotImplementedError

def _terminal(self) -> bool:
    return False

def _get_info(self) -> dict:
    return {} ``` ```python file=src/mjcourse/envs/reach.py """Reach: move arm7's `ee` site to a random target (Lessons 12.1 and 13.x).

Observation (float32, 23): joint positions (7), joint velocities (7), ee position (3, forward kinematics of the joint positions), target position (3), target minus ee (3). Action: see mjcourse.envs.arm (torque, joint_delta or ee_delta). Reward: minus the distance from ee to target (m) per step. Terminated: the ee is within success_radius of the target, unless terminate_on_success is False (reach and hold: every episode runs to the limit). Truncated: after max_episode_steps steps (0.05 s each by default). “””

from future import annotations

import mujoco import numpy as np from gymnasium import spaces

from mjcourse.envs.arm import ArmCommand from mjcourse.envs.base import MujocoEnv

TARGET_LOW, TARGET_HIGH = np.array([0.30, -0.30, 0.15]), np.array([0.60, 0.30, 0.50])

class ReachEnv(MujocoEnv): def init(self, action_mode: str = “joint_delta”, frame_skip: int = 25, max_episode_steps: int = 100, success_radius: float = 0.02, joint_noise: float = 0.1, terminate_on_success: bool = True, **kwargs): super().init(“reach”, frame_skip, max_episode_steps, **kwargs) self.arm = ArmCommand(self.model, self.data, action_mode, “ee”) self.action_space = self.arm.space self.observation_space = spaces.Box(-np.inf, np.inf, shape=(23,), dtype=np.float32) self.success_radius, self.joint_noise, self.terminate_on_success = success_radius, joint_noise, terminate_on_success

def _reset_task(self, options: dict) -> None:
    d = self.data
    d.qpos[:7] += self.np_random.uniform(-self.joint_noise, self.joint_noise, 7)
    d.mocap_pos[0] = options.get("target", self.np_random.uniform(TARGET_LOW, TARGET_HIGH))
    mujoco.mj_forward(self.model, d)
    self.arm.reset()

def _apply_action(self, action: np.ndarray) -> None:
    self.arm.apply(action)

def _control(self) -> None:
    self.arm.control()

def _distance(self) -> float:
    return float(np.linalg.norm(self.data.site("ee").xpos - self.data.mocap_pos[0]))

def _get_obs(self) -> np.ndarray:
    d = self.data
    ee, target = d.site("ee").xpos, d.mocap_pos[0]
    return np.concatenate([d.qpos[:7], d.qvel[:7], ee, target, target - ee]).astype(np.float32)

def _reward(self, action: np.ndarray) -> float:
    return -self._distance()

def _terminal(self) -> bool:
    return self.terminate_on_success and self._distance() < self.success_radius

def _get_info(self) -> dict:
    return {"distance": self._distance(), "is_success": self._distance() < self.success_radius} ``` ```python file=src/mjcourse/envs/arm.py """Three ways for a policy to command arm7 (Lesson 12.2).

“torque” the action is the actuator command: joint torques scaled to each actuator’s range. Nothing between the policy and the physics, gravity included. “joint_delta” the action moves joint targets by up to joint_step rad per environment step; PD with gravity compensation tracks them at the physics rate. “ee_delta” the action moves a target for a site by up to ee_step m per environment step (along the axes in ee_axes); a Cartesian impedance with the tool held pointing down tracks it (mjcourse.control.cartesian_impedance). With max_lead, the target never leads the site by more than that distance (m), which caps the force the impedance can apply (Lesson 10.3). The arm’s dofs and torque actuators are the first seven. “””

from future import annotations

import mujoco import numpy as np from gymnasium import spaces

from mjcourse import control, spatial

MODES = (“torque”, “joint_delta”, “ee_delta”)

class ArmCommand: def init(self, model: mujoco.MjModel, data: mujoco.MjData, mode: str, site: str, ee_axes: tuple[int, …] = (0, 1, 2), joint_step: float = 0.05, ee_step: float = 0.02, ee_low=(0.25, -0.35, 0.02), ee_high=(0.70, 0.35, 0.55), wn: float = 20.0, max_lead: float | None = None): if mode not in MODES: raise ValueError(f”mode must be one of {MODES}”) self.model, self.data, self.mode, self.site = model, data, mode, site self.ee_axes, self.joint_step, self.ee_step = list(ee_axes), joint_step, ee_step self.ee_low, self.ee_high = np.array(ee_low), np.array(ee_high) self.lo, self.hi = model.jnt_range[:7, 0], model.jnt_range[:7, 1] self.limit = model.actuator_ctrlrange[:7, 1] self.wn, self.max_lead = wn, max_lead size = {“torque”: 7, “joint_delta”: 7, “ee_delta”: len(self.ee_axes)}[mode] self.space = spaces.Box(-1.0, 1.0, shape=(size,), dtype=np.float32)

def reset(self) -> None:
    """Targets start where the arm is; call after the state is set."""
    d = self.data
    mass = np.zeros((self.model.nv, self.model.nv))
    mujoco.mj_fullM(self.model, d, mass)
    self.kp, self.kd = np.diag(mass)[:7] * self.wn**2, 2 * np.diag(mass)[:7] * self.wn
    self.q_target = d.qpos[:7].copy()
    self.x_target = d.site(self.site).xpos.copy()
    self.down = spatial.mat_to_quat(d.site(self.site).xmat.reshape(3, 3))
    self.q_rest = d.qpos[:7].copy()

def apply(self, action: np.ndarray) -> None:
    if self.mode == "torque":
        self.data.ctrl[:7] = action * self.limit
    elif self.mode == "joint_delta":
        self.q_target = np.clip(self.q_target + self.joint_step * action, self.lo, self.hi)
    else:
        self.x_target[self.ee_axes] += self.ee_step * action
        self.x_target = np.clip(self.x_target, self.ee_low, self.ee_high)
        if self.max_lead is not None:
            site = self.data.site(self.site).xpos
            lead = self.x_target - site
            norm = np.linalg.norm(lead[self.ee_axes])
            if norm > self.max_lead:
                self.x_target[self.ee_axes] = site[self.ee_axes] + lead[self.ee_axes] * self.max_lead / norm

def control(self) -> None:
    d = self.data
    if self.mode == "torque":
        return
    mujoco.mj_forward(self.model, d)
    if self.mode == "joint_delta":
        tau = self.kp * (self.q_target - d.qpos[:7]) - self.kd * d.qvel[:7] + d.qfrc_bias[:7]
        d.ctrl[:7] = np.clip(tau, -self.limit, self.limit)
    else:
        d.ctrl[:7] = control.cartesian_impedance(self.model, d, self.site, self.x_target, self.down, 2000.0, 300.0,
                                                 q_rest=self.q_rest, k_null=20.0) ``` The push task (`src/mjcourse/envs/push.py`) follows the same pattern; Lesson 12.2 embeds it.

```python file=examples/l12_1_gym_env.py “"”Lesson 12.1: the Gymnasium API written on MuJoCo, checked, broken on purpose, and timed.

INPUT mjcourse.envs: ReachEnv and PushEnv, each with three action modes PROCESS (1) gymnasium’s check_env on the six variants; (2) four broken variants: what check_env catches and what it does not; (3) the observation-is-a-view bug in a replay buffer; (4) stale kinematics: positions read after mj_step without mj_forward; (5) seeding: the same seed and actions reproduce an episode, reset() continues the stream; (6) terminated against truncated for a random and a scripted policy on Reach; (7) throughput: environment steps per second per action mode, and with vector environments OUTPUT printed tables

Run: python examples/l12_1_gym_env.py “””

import os import time import warnings

import gymnasium as gym import mujoco import numpy as np from gymnasium.utils.env_checker import check_env

import mjcourse.envs # noqa: F401 (registers mjcourse/Reach-v0 and mjcourse/Push-v0) from mjcourse.envs.reach import TARGET_HIGH, TARGET_LOW, ReachEnv

FAST = os.environ.get(“MJC_FAST”) == “1” MODES = (“torque”, “joint_delta”, “ee_delta”)

def checked(env) -> str: “"”Run check_env; return ‘passes’ or the error, plus warnings other than infinite bounds.””” with warnings.catch_warnings(record=True) as caught: warnings.simplefilter(“always”) try: check_env(env, skip_render_check=True) result = “passes” except Exception as error: # noqa: BLE001 - we report whatever the checker raises result = f”{type(error).name}: {error}” extra = [str(w.message).replace(“\x1b[33m”, “”).replace(“\x1b[0m”, “”) for w in caught if “infinity” not in str(w.message)] return result + (f” (warnings: {extra})” if extra else “”)

class Float64Obs(ReachEnv): def _get_obs(self): return super()._get_obs().astype(np.float64) # the space says float32

class GlobalRng(ReachEnv): def _reset_task(self, options): self.data.mocap_pos[0] = np.random.uniform(TARGET_LOW, TARGET_HIGH) # not self.np_random mujoco.mj_forward(self.model, self.data) self.arm.reset()

class ViewObs(ReachEnv): def init(self, kwargs): super().__init__(kwargs) self.observation_space = gym.spaces.Box(-np.inf, np.inf, shape=(7,), dtype=np.float64)

def _get_obs(self):
    return self.data.qpos[:7]                           # a view into MjData, not a copy

class NoTruncation(ReachEnv): def step(self, action): obs, reward, terminated, truncated, info = super().step(action) return obs, reward, terminated or truncated, False, info # time limit reported as termination

def part1_2() -> None: for env_id in (“mjcourse/Reach-v0”, “mjcourse/Push-v0”): for mode in MODES: env = gym.make(env_id, action_mode=mode).unwrapped print(f” {env_id:<18} {mode:<12} {checked(env)}”) env.close() print(“ broken on purpose:”) for cls, what in ((Float64Obs, “float64 observation in a float32 space”), (GlobalRng, “reset draws from np.random”), (ViewObs, “observation is a view of data.qpos”), (NoTruncation, “time limit reported as terminated”)): print(f” {what:<40} {checked(cls())}”)

def part3() -> None: env = ViewObs(action_mode=”torque”) buffer = [env.reset(seed=0)[0]] for _ in range(3): buffer.append(env.step(env.action_space.sample())[0]) same = all(np.array_equal(buffer[0], b) for b in buffer) print(f” 4 observations stored from a view: all equal {same}; they share memory with data.qpos: “ f”{np.shares_memory(buffer[0], env.data.qpos)}”) env = ReachEnv(action_mode=”torque”) buffer = [env.reset(seed=0)[0]] for _ in range(3): buffer.append(env.step(env.action_space.sample())[0]) print(f” 4 observations stored from copies: all equal {all(np.array_equal(buffer[0], b) for b in buffer)}”)

def part4() -> None: env = ReachEnv(action_mode=”torque”) m, d = env.model, env.data env.reset(seed=0) rng = np.random.default_rng(0) lag, speed = [], [] for _ in range(200): d.ctrl[:7] = rng.uniform(-1, 1, 7) * m.actuator_ctrlrange[:7, 1] * 0.3 for _ in range(env.frame_skip): mujoco.mj_step(m, d) stale = d.site(“ee”).xpos.copy() # computed at the start of the last step mujoco.mj_forward(m, d) lag.append(np.linalg.norm(d.site(“ee”).xpos - stale)) jac = np.zeros((3, m.nv)) mujoco.mj_jacSite(m, d, jac, None, m.site(“ee”).id) speed.append(np.linalg.norm(jac @ d.qvel)) lag, speed = np.array(lag), np.array(speed) print(f” ee position read after mj_step vs after mj_forward, 200 samples under random torques:”) print(f” difference mean {1000 * lag.mean():.2f} mm, max {1000 * lag.max():.2f} mm; “ f”ee speed times the timestep: mean {1000 * (speed * m.opt.timestep).mean():.2f} mm”)

def rollout(env, seed: int, actions: np.ndarray) -> np.ndarray: obs = [env.reset(seed=seed)[0]] for a in actions: obs.append(env.step(a)[0]) return np.array(obs)

def part5() -> None: actions = np.random.default_rng(1).uniform(-1, 1, (50, 7)).astype(np.float32) a, b = ReachEnv(), ReachEnv() same = np.array_equal(rollout(a, 3, actions), rollout(b, 3, actions)) other = np.array_equal(rollout(a, 3, actions), rollout(b, 4, actions)) print(f” seed 3 twice, same actions: identical {same}; seeds 3 and 4: identical {other}”) targets = [] for env in (ReachEnv(), ReachEnv()): env.reset(seed=7) targets.append([env.data.mocap_pos[0].copy()] + [env.reset()[0][17:20] for _ in range(3)]) print(f” reset(seed=7) then reset() three times, in two fresh environments: targets identical “ f”{all(np.allclose(x, y) for x, y in zip(*targets))}; the four targets differ from each other “ f”{len({tuple(np.round(t, 6)) for t in targets[0]}) == 4}”)

def episodes(env, policy, n: int, seed: int) -> dict: stats = {“terminated”: 0, “truncated”: 0, “length”: [], “return”: []} for i in range(n): obs, _ = env.reset(seed=seed + i) total, steps = 0.0, 0 while True: obs, reward, terminated, truncated, _ = env.step(policy(obs)) total, steps = total + reward, steps + 1 if terminated or truncated: stats[“terminated” if terminated else “truncated”] += 1 stats[“length”].append(steps) stats[“return”].append(total) break return stats

def part6() -> None: env = ReachEnv(action_mode=”ee_delta”) env.action_space.seed(100) # sample() uses the space’s own generator, not reset’s n = 20 if FAST else 100 random_policy = lambda obs: env.action_space.sample() # noqa: E731 toward = lambda obs: np.clip(obs[20:23] / 0.02, -1, 1).astype(np.float32) # noqa: E731 target minus ee, scaled print(f” {‘policy’:<22}{‘terminated’:>11}{‘truncated’:>10}{‘mean length’:>13}{‘mean return’:>13}”) for name, policy in ((“random”, random_policy), (“move toward target”, toward)): s = episodes(env, policy, n, seed=100) print(f” {name:<22}{s[‘terminated’]:>11d}{s[‘truncated’]:>10d}{np.mean(s[‘length’]):>13.1f}{np.mean(s[‘return’]):>13.2f}”)

def part7() -> None: n = 100 if FAST else 500 print(f” {‘environment’:<20}{‘mode’:<13}{‘steps/s’:>9}{‘physics steps/s’:>17}”) for env_id in (“mjcourse/Reach-v0”, “mjcourse/Push-v0”): for mode in MODES: env = gym.make(env_id, action_mode=mode) env.reset(seed=0) start = time.perf_counter() for _ in range(n): _, _, terminated, truncated, _ = env.step(env.action_space.sample()) if terminated or truncated: env.reset() rate = n / (time.perf_counter() - start) print(f” {env_id:<20}{mode:<13}{rate:>9.0f}{rate * env.unwrapped.frame_skip:>17.0f}”) env.close() print(f” vector environments, Reach joint_delta ({os.cpu_count()} CPU cores):”) for count in ((1, 4) if FAST else (1, 4, 16, 32)): envs = gym.vector.AsyncVectorEnv([lambda: gym.make(“mjcourse/Reach-v0”, action_mode=”joint_delta”)] * count) envs.reset(seed=0) start = time.perf_counter() for _ in range(n // 5): envs.step(envs.action_space.sample()) rate = count * (n // 5) / (time.perf_counter() - start) print(f” {count:>3d} processes: {rate:>8.0f} environment steps/s”) envs.close()

if name == “main”: print(f”MuJoCo {mujoco.version}, gymnasium {gym.version}”) print(“(1, 2) gymnasium.utils.env_checker.check_env (render check skipped)”) part1_2() print(“(3) a replay buffer filled from an observation that is a view”) part3() print(“(4) stale kinematics after mj_step”) part4() print(“(5) seeding”) part5() print(“(6) terminated or truncated: Reach with ee_delta actions, 100 steps of 0.05 s”) part6() print(“(7) throughput, single process”) part7()

Output:

```text
MuJoCo 3.14.0, gymnasium 1.3.0
(1, 2) gymnasium.utils.env_checker.check_env (render check skipped)
  mjcourse/Reach-v0  torque       passes
  mjcourse/Reach-v0  joint_delta  passes
  mjcourse/Reach-v0  ee_delta     passes
  mjcourse/Push-v0   torque       passes
  mjcourse/Push-v0   joint_delta  passes
  mjcourse/Push-v0   ee_delta     passes
  broken on purpose:
  float64 observation in a float32 space   AssertionError: The first element returned by `env.reset()` is not within the observation space.
  reset draws from np.random               AssertionError: Deterministic step observations are not equivalent for the same seed and action
  observation is a view of data.qpos       passes
  time limit reported as terminated        passes
(3) a replay buffer filled from an observation that is a view
  4 observations stored from a view: all equal True; they share memory with data.qpos: True
  4 observations stored from copies: all equal False
(4) stale kinematics after mj_step
  ee position read after mj_step vs after mj_forward, 200 samples under random torques:
  difference mean 1.77 mm, max 5.89 mm; ee speed times the timestep: mean 1.77 mm
(5) seeding
  seed 3 twice, same actions: identical True; seeds 3 and 4: identical False
  reset(seed=7) then reset() three times, in two fresh environments: targets identical True; the four targets differ from each other True
(6) terminated or truncated: Reach with ee_delta actions, 100 steps of 0.05 s
  policy                 terminated truncated  mean length  mean return
  random                          2        98         98.1       -27.13
  move toward target            100         0         10.7        -1.31
(7) throughput, single process
  environment         mode           steps/s  physics steps/s
  mjcourse/Reach-v0   torque            5831           145775
  mjcourse/Reach-v0   joint_delta       2919            72987
  mjcourse/Reach-v0   ee_delta           465            11636
  mjcourse/Push-v0    torque            2314            57848
  mjcourse/Push-v0    joint_delta       1209            30226
  mjcourse/Push-v0    ee_delta           356             8896
  vector environments, Reach joint_delta (48 CPU cores):
      1 processes:     1303 environment steps/s
      4 processes:     3499 environment steps/s
     16 processes:    11838 environment steps/s
     32 processes:    12009 environment steps/s

Debugging

Every observation in the buffer is the same. The observation is a view of MjData (data.qpos[:7], data.site("ee").xpos). Concatenate or .copy() before returning.

The end effector in the observation disagrees with the joint angles by millimetres. Positions read after mj_step without mj_forward.

check_env fails with “Deterministic step observations are not equivalent”. Randomness from a generator other than self.np_random: np.random, Python’s random, or a generator created once at import.

Training curves that change with the time limit. Truncation reported as termination, or the reverse. Print a few transitions at the time limit and check the two flags.

Runs are too slow. Profile the step: the inner controller usually costs more than mj_step. Lower its rate, simplify it, or move to vector environments.

Exercise

Add a _get_info entry with the minimum distance reached so far in the episode, and check with check_env that the environment still passes. Then write the test from the recommendation that fails on NoTruncation and passes on ReachEnv.

Challenge

Make the end-effector mode four times faster without changing what it does: run the Cartesian impedance’s null-space projection once per environment step instead of once per physics step, keep the rest at 500 Hz, and show with a paired comparison on 50 seeded episodes of the scripted policy that the trajectories differ by less than 1 mm.

Research connection

[!research] The environment is part of the method Results reported on “the same” task depend on control frequency, action space, observation content, time limits and how truncation is handled, and these differ between implementations of nominally identical benchmarks (inference; Lesson 12.2 measures the effect of the action space). Treat the environment’s version, its parameters and its source as part of an experiment’s specification, the same way Level 21 treats seeds and evaluation protocol. The time-limit question in particular was studied by Pardo et al. (ICML 2018, arXiv:1712.00378): when time limits only serve training, the learner should bootstrap from the value of the last state instead of treating it as terminal; when the limit is part of the task, the remaining time belongs in the observation. Ignoring both causes state aliasing and training instability in their experiments.

{"id": "12.1-check", "title": "Knowledge check", "questions": [
  {"kind": "numeric", "q": "A model has a 2 ms timestep and the environment holds each action for 25 physics steps. At what rate (Hz) does the policy act?",
   "answer": 20, "tol": 0, "unit": "Hz",
   "explain": "<p>$\\Delta t = 25 \\times 2$ ms = 50 ms, so 20 Hz.</p>"},
  {"kind": "mcq", "q": "An episode of the reach task ends because 100 steps have passed and the arm is still 0.3 m from the target. What should step return?",
   "options": ["terminated=True, truncated=False", "terminated=False, truncated=True", "terminated=True, truncated=True", "terminated=False, truncated=False"],
   "answer": 1,
   "explain": "<p>Time ran out; the task did not end. A learner will bootstrap the value of that state.</p>"},
  {"kind": "mcq", "q": "Which defect does gymnasium's check_env not detect?",
   "options": ["A float64 observation in a float32 space", "Randomness drawn from np.random", "An observation that is a view of data.qpos", "An observation outside the declared shape"],
   "answer": 2,
   "explain": "<p>Output (2): the view passes the checker, and a replay buffer filled from it holds one state many times (output 3).</p>"},
  {"kind": "numeric", "q": "The end effector moves at 0.9 m/s and the timestep is 2 ms. By about how many millimetres is its position stale if read after mj_step without mj_forward?",
   "answer": 1.8, "tol": 0.05, "unit": "mm",
   "explain": "<p>One timestep of motion: 0.9 m/s × 2 ms = 1.8 mm (output 4 measured a mean of 1.77 mm).</p>"}
]}

Next

Lesson 12.2 asks what to put in those spaces: torque, joint-target or end-effector actions; dense, shaped, sparse or careless rewards; and observations a real robot could actually provide.