A benchmark is a set of choices with a name on it. Any of those choices can change between versions while the name stays almost the same:
This lesson reads benchmarks along those seven axes, and measures how far apart two versions of “the same” environments are: Gymnasium’s MuJoCo locomotion and control tasks in versions 4 and 5, run side by side on the same MuJoCo release.
[!established] Seven questions to ask of any benchmark
Axis Question task which model (file and version), which objective, which variations observation which quantities, in which order, with what noise, from which state (Lesson 20.1: positions read after mj_stepare one step old)action which space (torque, position target, end-effector delta), its limits, the control period episode the time limit, the termination conditions, and whether a time limit is reported as truncation (Lesson 13.1) initialization the distribution of initial states, and their seeds evaluation which starts or seeds, how many episodes, train against held-out splits, deterministic or stochastic policy metric return or success, its exact definition, and its aggregation across seeds and tasks (Lesson 21.2)
There is no lab for this lesson; the experiment needs Gymnasium in Python. Think of each benchmark version as a row in the tables below. Two rows can share a task name and differ in every other column.
Gymnasium 1.3.0 ships each MuJoCo task in several versions; versions 4 and 5 both run on MuJoCo 3.14.0. As specified:
| Task | Observation v4 / v5 | Actions | Control period | Time limit | Same compiled model |
|---|---|---|---|---|---|
| Hopper | 11 / 11 | 3 | 0.008 s | 1000 | yes |
| Walker2d | 17 / 17 | 6 | 0.008 s | 1000 | no |
| HalfCheetah | 17 / 17 | 6 | 0.050 s | 1000 | yes |
| Ant | 27 / 105 | 8 | 0.050 s | 1000 | yes |
| Humanoid | 376 / 348 | 17 | 0.015 s | 1000 | yes |
| Swimmer | 8 / 8 | 2 | 0.040 s | 1000 | yes |
| Reacher | 11 / 10 | 2 | 0.020 s | 50 | yes |
| InvertedPendulum | 4 / 4 | 1 | 0.040 s | 1000 | yes |
“Same compiled model” compares mjcourse.experiment.model_fingerprint of the two environments’ models, which covers every number of the compiled model. Run with uniform random actions and with zero actions, 50 seeds each, the versions give these mean episode lengths and returns:
| Task | random v4 | random v5 | zero v4 | zero v5 |
|---|---|---|---|---|
| Hopper | 23.3 steps, 19.1 | 23.3 steps, 18.1 | 152.2 steps, 152.0 | 152.2 steps, 151.0 |
| Walker2d | 22.2 steps, 3.4 | 21.9 steps, 2.2 | 109.2 steps, 94.1 | 115.5 steps, 93.6 |
| Ant | 216.8 steps, −75.6 | 216.8 steps, −76.8 | 1000 steps, 1000.2 | 1000 steps, 994.2 |
| Humanoid | 23.1 steps, 116.8 | 23.1 steps, 108.7 | 40.3 steps, 207.9 | 40.3 steps, 198.7 |
| InvertedPendulum | 6.8 steps, 6.8 | 6.8 steps, 5.8 | 27.0 steps, 27.0 | 27.0 steps, 26.0 |
HalfCheetah, Swimmer and Reacher gave identical numbers in both versions for these two policies. The reset distributions are identical too: the mean standard deviation of the initial qpos and qvel over 200 resets matches to five decimals for every task, and the stored termination ranges (Hopper’s healthy height 0.7 and up, Walker2d’s 0.8 to 2.0, and so on) are the same.
Each difference has a cause, and Gymnasium’s own version history in the installed source names it:
None of these is visible in a results table that says “Hopper” or “Ant”. A return on Humanoid differs by about 8 to 9 between the versions for the same behaviour here, and an observation space can change size.
Manipulation benchmarks built on MuJoCo and other simulators have the same structure and the same hazard. Meta-World’s task definitions changed between releases, CALVIN’s splits (training on environments A, B and C and testing on D, against training on all four) answer different questions, LIBERO-PRO and LIBERO-Plus perturb LIBERO’s initial states and instructions to test what the original protocol could not, and SimplerEnv was built to measure how well simulated success predicts real success. The reading list’s benchmark section names each paper and the protocol question to ask of it.
[!recommendation] Before comparing two numbers Check, for both:
- the environment id with its version;
- the library and MuJoCo versions;
- a fingerprint of the compiled model;
- the observation and action spaces;
- the control period and the time limit;
- the reward terms and termination;
- the reset distribution;
- the evaluation protocol, meaning seeds, episodes and splits.
If any differ, the numbers come from different benchmarks; say so in the table, or rerun the baseline. When you build a benchmark, version it, write each of the seven axes into its documentation, and record them in every run (Lesson 21.1).
INPUT: Gymnasium's MuJoCo environments v4 and v5 (gymnasium 1.3.0, MuJoCo 3.14.0)
PROCESS: spaces, control period, time limit and model fingerprint per version; episodes under random and zero actions; reset spread; termination settings
OUTPUT: printed tables
```python file=examples/l21_3_benchmark_anatomy.py “"”Lesson 21.3: one task name, two benchmarks: Gymnasium’s MuJoCo environments v4 against v5 (Level 21).
INPUT Gymnasium’s MuJoCo environments in versions 4 and 5 (the installed gymnasium release, MuJoCo 3.14.0): Hopper, Walker2d, HalfCheetah, Ant, Humanoid, Swimmer, Reacher, InvertedPendulum PROCESS (1) for each pair: observation and action sizes, control timestep, frame skip, time limit, and whether the compiled models are identical (mjcourse.experiment.model_fingerprint); (2) episodes under uniform random actions and under zero actions, 50 seeds each: mean length and return; (3) the initial-state distribution: the standard deviation of qpos and qvel over 200 resets; (4) the termination settings the two versions store (healthy z range, termination on unhealthy states) OUTPUT printed tables
Run: python examples/l21_3_benchmark_anatomy.py (about 2 minutes) “””
import os import warnings
import gymnasium as gym import numpy as np
from mjcourse.experiment import model_fingerprint
warnings.filterwarnings(“ignore”) # v4 environments announce their deprecation FAST = os.environ.get(“MJC_FAST”) == “1” TASKS = (“Hopper”, “Walker2d”, “HalfCheetah”, “Ant”, “Humanoid”, “Swimmer”, “Reacher”, “InvertedPendulum”) EPISODES = 5 if FAST else 50
def episodes(env_id: str, policy: str) -> tuple[float, float]: env = gym.make(env_id) lengths, returns = [], [] for seed in range(EPISODES): obs, _ = env.reset(seed=seed) env.action_space.seed(seed) done, steps, total = False, 0, 0.0 while not done: action = env.action_space.sample() if policy == “random” else np.zeros(env.action_space.shape) obs, reward, terminated, truncated, _ = env.step(action) total, steps, done = total + reward, steps + 1, terminated or truncated lengths.append(steps) returns.append(total) env.close() return float(np.mean(lengths)), float(np.mean(returns))
def reset_spread(env_id: str, n: int) -> tuple[float, float]: env = gym.make(env_id) qpos, qvel = [], [] for seed in range(n): env.reset(seed=seed) qpos.append(env.unwrapped.data.qpos.copy()) qvel.append(env.unwrapped.data.qvel.copy()) env.close() return float(np.std(qpos, axis=0).mean()), float(np.std(qvel, axis=0).mean())
if name == “main”: print(f”gymnasium {gym.version}; {EPISODES} episodes per row”) print(“(1) the environments as specified”) print(f” {‘task’:<18}{‘obs v4/v5’:>11}{‘act’:>5}{‘dt v4/v5’:>14}{‘frame skip’:>12}{‘time limit’:>12} same compiled model”) for task in TASKS: envs = [gym.make(f”{task}-v{v}”) for v in (4, 5)] u = [e.unwrapped for e in envs] same = model_fingerprint(u[0].model) == model_fingerprint(u[1].model) print(f” {task:<18}{envs[0].observation_space.shape[0]:>5}/{envs[1].observation_space.shape[0]:<5}” f”{envs[0].action_space.shape[0]:>5}{u[0].dt:>8.3f}/{u[1].dt:<6.3f}{u[0].frame_skip:>6}/{u[1].frame_skip:<5}” f”{envs[0].spec.max_episode_steps:>6}/{envs[1].spec.max_episode_steps:<5} {‘yes’ if same else ‘no’}”) for e in envs: e.close() print(“(2) mean episode length and return, uniform random actions and zero actions”) print(f” {‘task’:<18}{‘random v4’:>22}{‘random v5’:>22}{‘zero v4’:>22}{‘zero v5’:>22}”) for task in TASKS: cells = [] for policy in (“random”, “zero”): for v in (4, 5): length, ret = episodes(f”{task}-v{v}”, policy) cells.append(f”{length:7.1f} steps {ret:9.1f}”) print(f” {task:<18}” + ““.join(f”{c:>22}” for c in cells)) print(“(3) initial states: mean standard deviation over 200 resets, qpos and qvel”) for task in TASKS: cells = [reset_spread(f”{task}-v{v}”, 20 if FAST else 200) for v in (4, 5)] print(f” {task:<18} v4 {cells[0][0]:.5f}, {cells[0][1]:.5f} v5 {cells[1][0]:.5f}, {cells[1][1]:.5f}”) print(“(4) termination settings stored by the environments”) for task in (“Hopper”, “Walker2d”, “Ant”, “Humanoid”): cells = [] for v in (4, 5): u = gym.make(f”{task}-v{v}”).unwrapped z = getattr(u, “_healthy_z_range”, None) cells.append(f”v{v}: healthy z {z}, terminate when unhealthy {getattr(u, ‘_terminate_when_unhealthy’, None)}”) print(f” {task:<10} “ + “; “.join(cells))
Output:
```text
gymnasium 1.3.0; 50 episodes per row
(1) the environments as specified
task obs v4/v5 act dt v4/v5 frame skip time limit same compiled model
Hopper 11/11 3 0.008/0.008 4/4 1000/1000 yes
Walker2d 17/17 6 0.008/0.008 4/4 1000/1000 no
HalfCheetah 17/17 6 0.050/0.050 5/5 1000/1000 yes
Ant 27/105 8 0.050/0.050 5/5 1000/1000 yes
Humanoid 376/348 17 0.015/0.015 5/5 1000/1000 yes
Swimmer 8/8 2 0.040/0.040 4/4 1000/1000 yes
Reacher 11/10 2 0.020/0.020 2/2 50/50 yes
InvertedPendulum 4/4 1 0.040/0.040 2/2 1000/1000 yes
(2) mean episode length and return, uniform random actions and zero actions
task random v4 random v5 zero v4 zero v5
Hopper 23.3 steps 19.1 23.3 steps 18.1 152.2 steps 152.0 152.2 steps 151.0
Walker2d 22.2 steps 3.4 21.9 steps 2.2 109.2 steps 94.1 115.5 steps 93.6
HalfCheetah 1000.0 steps -277.6 1000.0 steps -277.6 1000.0 steps -0.1 1000.0 steps -0.1
Ant 216.8 steps -75.6 216.8 steps -76.8 1000.0 steps 1000.2 1000.0 steps 994.2
Humanoid 23.1 steps 116.8 23.1 steps 108.7 40.3 steps 207.9 40.3 steps 198.7
Swimmer 1000.0 steps 3.8 1000.0 steps 3.8 1000.0 steps 1.6 1000.0 steps 1.6
Reacher 50.0 steps -42.8 50.0 steps -42.8 50.0 steps -11.5 50.0 steps -11.5
InvertedPendulum 6.8 steps 6.8 6.8 steps 5.8 27.0 steps 27.0 27.0 steps 26.0
(3) initial states: mean standard deviation over 200 resets, qpos and qvel
Hopper v4 0.00288, 0.00296 v5 0.00288, 0.00296
Walker2d v4 0.00290, 0.00297 v5 0.00290, 0.00297
HalfCheetah v4 0.05796, 0.09889 v5 0.05796, 0.09889
Ant v4 0.05882, 0.09754 v5 0.05882, 0.09754
Humanoid v4 0.00587, 0.00575 v5 0.00587, 0.00575
Swimmer v4 0.05694, 0.05955 v5 0.05694, 0.05955
Reacher v4 0.08048, 0.00147 v5 0.08048, 0.00147
InvertedPendulum v4 0.00580, 0.00568 v5 0.00580, 0.00568
(4) termination settings stored by the environments
Hopper v4: healthy z (0.7, inf), terminate when unhealthy True; v5: healthy z (0.7, inf), terminate when unhealthy True
Walker2d v4: healthy z (0.8, 2.0), terminate when unhealthy True; v5: healthy z (0.8, 2.0), terminate when unhealthy True
Ant v4: healthy z (0.2, 1.0), terminate when unhealthy True; v5: healthy z (0.2, 1.0), terminate when unhealthy True
Humanoid v4: healthy z (1.0, 2.0), terminate when unhealthy True; v5: healthy z (1.0, 2.0), terminate when unhealthy True
Your reproduction scores lower than the paper by a constant. Check the version: a reward term, such as the survival reward on the terminal step, may differ.
A trained policy cannot be loaded into the new environment. The observation space changed size between versions (Ant, Humanoid, Reacher).
Two runs of “the same” environment disagree with the same seeds. Compare model fingerprints and library versions; Walker2d’s feet changed friction between versions.
A baseline from a paper cannot be matched at all. The paper may not say which version, split or protocol it used. Write to the authors, or report the mismatch.
Train or tune one policy on Hopper-v4, then evaluate it on Hopper-v5 without retraining. How much of the difference in return does the terminal-step reward explain, and what is left?
Pick one manipulation benchmark from the reading list and write its seven-axis description from its paper and code, for the version you would cite. List every place where a choice is not stated, and design the smallest experiment that would show whether that choice changes a reported number.
[!research] Reporting benchmark results Towers et al. (Gymnasium, arXiv:2407.17032, in the reading list) present the library as a standard interface with tools for reproducible research. The version suffix of an environment id works as one such tool: as measured here, v4 and v5 coexist and differ, so a version change is a warning that numbers across it are not comparable (inference). Every results table in a paper should carry the version, and every comparison against a published number should check it first.
{"id": "21.3-check", "title": "Knowledge check", "questions": [
{"kind": "mcq", "q": "InvertedPendulum's zero-action episodes last 27 steps in both versions but return 27.0 in v4 and 26.0 in v5. Why?",
"options": ["Different physics", "v5 no longer pays the survival reward on the step where the pendulum falls", "Different seeds", "Different time limit"],
"answer": 1,
"explain": "<p>The v5 version history: the healthy reward is only given while healthy.</p>"},
{"kind": "mcq", "q": "Which task's compiled model differs between v4 and v5?",
"options": ["Hopper", "Walker2d", "Ant", "Reacher"],
"answer": 1,
"explain": "<p>v5 gives both feet the same friction; v4 had 1.9 and 0.9.</p>"},
{"kind": "mcq", "q": "A paper reports Ant results without a version. What can you compare it with?",
"options": ["Any Ant result", "Nothing, until the version and protocol are known: the observation space alone differs, 27 against 105", "Only v5", "Only v4"],
"answer": 1,
"explain": "<p>The reward also differs (contact cost), so returns are not comparable across versions.</p>"},
{"kind": "mcq", "q": "Which of the seven axes does 'positions read after mj_step are one step old' belong to?",
"options": ["task", "observation", "metric", "initialization"],
"answer": 1,
"explain": "<p>It changes what the agent observes; Gymnasium's v5 notes fix exactly such an off-by-one-step position in info.</p>"}
]}
Lesson 21.4 uses MuJoCo as a model to plan with, and asks what changes when the model is learned.