An environment used for research is a measuring instrument, and a number it produces means only what its specification says. The specification covers the task, the distribution of initial states, the definition of success, the horizon, and the evaluation protocol: which starts, how many, how a policy is chosen, and what is reported. Write it down before running anything, state the result that would falsify the hypothesis, and let every run record what a reviewer will ask for. This lesson does that for the point-mass task of Level 14, using a small module added to the course library (mjcourse.experiment), and measures three mistakes that quietly change results: choosing on the evaluation set, letting the success definition drift, and editing the model in code where no file hash can see it.

Visual intuition

There is no lab for this lesson. The Lesson 14.1 lab shows the task; this lesson is about what is written around it.

Mathematics

Choosing on the evaluation set

Eight BC policies come from the same 20 demonstrations and differ only in the training seed. Each is scored with pushes on 50 development starts and on 400 fresh starts:

The seed that wins on the development set, seed 7, scored 0.72 there and 0.655 on fresh starts. Reporting its development score would overstate it by 0.065, because picking the maximum of eight noisy scores picks noise too. The protocol avoids that: choose on development starts, report on held-out ones. Here the held-out interval for seed 7 is about [0.61, 0.70], so the hypothesis survives. The table also shows a second warning: the same data and code give anything from 0.333 to 0.685 depending on the training seed (mean 0.549). A single seed is an anecdote; Lesson 21.2 deals with seeds.

The definition of success

The same policies, run for the full 6 s on the 400 held-out starts without pushes, scored three ways:

The policy cloned from 80 demonstrations passes through the goal on every start but stops there on only 78% of them. “Success” defined as reaching would report 1.00 for a policy that often fails the task, and its interval would say nothing about the mismatch. The success definition belongs in the protocol, in code, with the task’s actual requirement (here, stopping). (The 80-demonstration policy stops less often than the 5-demonstration one. Lesson 14.1 shows how regression averages the expert’s two branches at y = 0, a plausible cause here (inference).)

A model edited in code

Item	For the point-mass study here
hypothesis	BC from 20 clean demonstrations, chosen among 8 training seeds on a development set, succeeds in at least 0.6 of held-out episodes with random pushes (1 per second)
task and model	point mass around an obstacle (Level 14), `point_mass.xml`, recorded by file hash and compiled-model fingerprint
initial states	uniform in x from −0.40 to −0.30 m and y from −0.25 to 0.25 m, at rest; training, development and held-out starts from separate seeds
success	within 4 cm of the goal and slower than 5 cm/s at some step within 6 s
selection	the policy is chosen on 50 development starts, never on the held-out ones
metric	success on 400 held-out starts with a 95% Wilson interval
falsified if	the held-out interval lies entirely below 0.6

Training seed	0	1	2	3	4	5	6	7
development (50 starts)	0.58	0.62	0.36	0.64	0.50	0.50	0.48	0.72
fresh (400 starts)	0.595	0.655	0.333	0.685	0.480	0.367	0.620	0.655

Policy	stopped at the goal	within 4 cm at any moment	within 4 cm at 6 s
expert	1.00 [0.99, 1.00]	1.00 [0.99, 1.00]	1.00 [0.99, 1.00]
BC, 5 demonstrations	0.98 [0.96, 0.99]	0.99 [0.97, 0.99]	0.94 [0.91, 0.96]
BC, 80 demonstrations	0.78 [0.73, 0.82]	1.00 [0.99, 1.00]	0.78 [0.73, 0.82]

Doubling the point mass’s damping in code after loading leaves the XML file unchanged: its SHA-256 stays 958d113c98c0fdbc… before and after. The compiled model’s fingerprint, a hash over every numeric array of mjModel and its options, changes from 9f1604a7f673cc6a… to 89ee39add925882d…. The expert’s success is unchanged (1.00), the 5-demonstration BC policy’s moves from 0.978 to 0.958. Without the fingerprint, the second number would be reported as if it came from the first model. Edits like this are common: domain randomization (Lesson 17.1), system identification (Level 18), and quick experiments all change models after loading.

The record

mjcourse.experiment.run_record writes the protocol, the software versions (Python, platform, MuJoCo, NumPy, Gymnasium, PyTorch, SciPy, and the git commit when run in a repository), each model’s file hash and fingerprint, the configuration and the results into one JSON file per run. The script’s record is runs/l21_1_record.json.

Implementation

INPUT: the point-mass task and its expert (Level 14), behaviour cloning, mjcourse.experiment
PROCESS: a protocol written first; selection among 8 training seeds on development starts against fresh starts; three success definitions; a model edit seen by the fingerprint and not by the file hash; a run record
OUTPUT: printed tables and runs/l21_1_record.json

```python file=src/mjcourse/experiment.py “"”Experiment records for MuJoCo studies (Level 21): what a reviewer will ask for, written by code.

A study states its protocol before it runs (Protocol), and every run writes a record with the protocol, the software versions, a hash of each model file and a fingerprint of each compiled model, the configuration and the results (run_record). The fingerprint covers the compiled model’s numbers, so it also changes when code edits a model after loading it, which a hash of the XML file cannot see. “””

import dataclasses import hashlib import json import platform import subprocess import sys import time from importlib import metadata from pathlib import Path

@dataclasses.dataclass(frozen=True) class Protocol: “"”Everything decided before the first run. Fields are free text where a number would hide a choice.”””

name: str
hypothesis: str
task: str
initial_states: str
success: str
metric: str
train_seeds: tuple[int, ...] = ()
eval_seeds: tuple[int, ...] = ()
falsified_if: str = ""

def file_sha256(path: str | Path) -> str: return hashlib.sha256(Path(path).read_bytes()).hexdigest()

def model_fingerprint(model: mujoco.MjModel) -> str: “"”SHA-256 over every numeric array of the compiled model and its options.””” digest = hashlib.sha256() for name in sorted(dir(model)): if name.startswith(“”): continue try: value = getattr(model, name) except Exception: continue if isinstance(value, np.ndarray): digest.update(name.encode()) digest.update(np.ascontiguousarray(value).tobytes()) for name in sorted(n for n in dir(model.opt) if not n.startswith(“”)): value = getattr(model.opt, name) if isinstance(value, (int, float, np.ndarray)): digest.update(name.encode()) digest.update(np.asarray(value).tobytes()) return digest.hexdigest()

def _version(package: str) -> str | None: try: return metadata.version(package) except metadata.PackageNotFoundError: return None

def software() -> dict: “"”Versions a result depends on, and the code’s git commit when run inside a repository.””” try: commit = subprocess.run([“git”, “rev-parse”, “HEAD”], capture_output=True, text=True, timeout=5).stdout.strip() or None except (OSError, subprocess.TimeoutExpired): commit = None return {“python”: sys.version.split()[0], “platform”: platform.platform(), “mujoco”: mujoco.version, “numpy”: np.version, **{p: _version(p) for p in (“gymnasium”, “torch”, “scipy”)}, “git_commit”: commit}

def run_record(protocol: Protocol, models: dict[str, tuple[mujoco.MjModel, str | Path | None]], config: dict, results: dict) -> dict: “"”models maps a name to (compiled model, its XML file or None).””” return {“protocol”: dataclasses.asdict(protocol), “software”: software(), “models”: {name: {“file_sha256”: file_sha256(path) if path else None, “fingerprint”: model_fingerprint(m), “timestep”: m.opt.timestep, “integrator”: mujoco.mjtIntegrator(m.opt.integrator).name} for name, (m, path) in models.items()}, “config”: config, “results”: results, “written”: time.strftime(“%Y-%m-%dT%H:%M:%S%z”)}

def save(record: dict, path: str | Path) -> Path: path = Path(path) path.parent.mkdir(parents=True, exist_ok=True) path.write_text(json.dumps(record, indent=2, default=float)) return path

```python file=examples/l21_1_environment_design.py
"""Lesson 21.1: designing a research-grade environment, and three mistakes that invalidate results (Level 21).

INPUT   the point-mass task of Level 14 (start left of a round obstacle, stop within 4 cm of the goal and slower
        than 5 cm/s, within 6 s), its scripted expert, and behaviour cloning (mjcourse.il.bc)
PROCESS (0) a protocol written before running, and the record each run saves (mjcourse.experiment);
        (1) choosing on the evaluation set: 8 BC policies fitted on the same 20 demonstrations with different
            training seeds, scored on 50 development starts with random pushes (1 per second, Level 14); the best is
            then scored on 400 fresh starts with fresh pushes;
        (2) the success definition: the same policies, run for the full 6 s, scored as "stopped at the goal"
            (the task's definition), "within 4 cm at any moment", and "within 4 cm when time runs out";
        (3) a silent model change: the point mass's damping doubled in code after loading; the XML file's hash,
            the compiled model's fingerprint, and the expert's success before and after
OUTPUT  printed tables and a run record in ./runs/l21_1_record.json

Run:  python examples/l21_1_environment_design.py          (about 1 minute)
"""
# requires: torch

import json
import os

import mujoco
import numpy as np

from mjcourse import experiment, model_path
from mjcourse.il import bc, pointmass
from mjcourse.stats import wilson_interval

FAST = os.environ.get("MJC_FAST") == "1"
HELD_OUT = 100 if FAST else 400


def full_rollout(sim: pointmass.PointMass, policy, start: np.ndarray) -> np.ndarray:
    """(T, 4) states over the whole 6 s, without stopping at success."""
    m, d = sim.model, sim.data
    mujoco.mj_resetData(m, d)
    d.qpos[:] = start
    mujoco.mj_forward(m, d)
    states = []
    for _ in range(pointmass.MAX_STEPS):
        s = np.r_[d.qpos, d.qvel]
        d.ctrl[:] = np.clip(np.asarray(policy(s), dtype=float), -1.0, 1.0)
        mujoco.mj_step(m, d)
        states.append(np.r_[d.qpos, d.qvel])
    return np.array(states)


def definitions(states: np.ndarray) -> tuple[bool, bool, bool]:
    dist = np.linalg.norm(states[:, :2] - pointmass.GOAL, axis=1)
    speed = np.linalg.norm(states[:, 2:], axis=1)
    return bool(np.any((dist < 0.04) & (speed < 0.05))), bool(np.any(dist < 0.04)), bool(dist[-1] < 0.04)


def rate(flags) -> str:
    flags = list(flags)
    lo, hi = wilson_interval(int(sum(flags)), len(flags))
    return f"{np.mean(flags):.2f} [{lo:.2f}, {hi:.2f}]"


if __name__ == "__main__":
    protocol = experiment.Protocol(
        name="bc-held-out-starts",
        hypothesis="Behaviour cloning from 20 clean demonstrations, chosen among 8 training seeds on a development set, "
                   "succeeds in at least 0.6 of held-out episodes with random pushes (1 per second).",
        task="point mass: from the left of a round obstacle, stop at the goal on its right (Level 14)",
        initial_states="positions uniform in x [-0.40, -0.30] m, y [-0.25, 0.25] m, at rest",
        success="within 4 cm of the goal and slower than 5 cm/s at some step within 6 s (600 steps of 10 ms)",
        metric="success rate on 400 held-out starts with pushes and a 95% Wilson interval; the policy is chosen on "
               "50 separate development starts",
        train_seeds=(1,), eval_seeds=(2,),
        falsified_if="the held-out interval lies entirely below 0.6")
    print(f"MuJoCo {mujoco.__version__}; protocol '{protocol.name}' written before the runs")
    sim = pointmass.PointMass()
    held_out = pointmass.sample_starts(np.random.default_rng(2), HELD_OUT)
    print("(1) eight training seeds, one data set: success with pushes on 50 development starts and on fresh starts")
    train = pointmass.sample_starts(np.random.default_rng(1), 20)
    demos = [sim.rollout(pointmass.expert, s) for s in train]
    x, y = np.vstack([r.states for r in demos]), np.vstack([r.actions for r in demos])
    dev = pointmass.sample_starts(np.random.default_rng(3), 50)
    scores, results = [], {}
    for seed in range(2 if FAST else 8):
        policy, _ = bc.fit(x, y, steps=500 if FAST else 3000, seed=seed)
        d = pointmass.evaluate(policy, dev, push_rate=1.0, seed=11)["success"]
        f = pointmass.evaluate(policy, held_out, push_rate=1.0, seed=12)["success"]
        scores.append((d, f))
        print(f"    training seed {seed}: development {d:.2f}, fresh {f:.3f}")
    scores = np.array(scores)
    best = int(np.argmax(scores[:, 0]))
    results["selection"] = {"selected_seed": best, "dev": float(scores[best, 0]), "fresh": float(scores[best, 1]),
                            "fresh_mean": float(scores[:, 1].mean())}
    lo, hi = wilson_interval(round(scores[best, 1] * HELD_OUT), HELD_OUT)
    verdict = "falsified" if hi < 0.6 else "not falsified"
    print(f"    chosen on the development set: seed {best}, which scored {scores[best, 0]:.2f} there and {scores[best, 1]:.3f} on "
          f"{HELD_OUT} fresh starts; the fresh scores of all seeds range from {scores[:, 1].min():.3f} to {scores[:, 1].max():.3f} "
          f"(mean {scores[:, 1].mean():.3f})")
    print(f"    the protocol's test: held-out interval [{lo:.3f}, {hi:.3f}] against 0.6: hypothesis {verdict}")
    policies = {}
    for n in ((5, 80) if not FAST else (5,)):
        starts = pointmass.sample_starts(np.random.default_rng(1), n)
        runs = [sim.rollout(pointmass.expert, s) for s in starts]
        policies[n], _ = bc.fit(np.vstack([r.states for r in runs]), np.vstack([r.actions for r in runs]),
                                steps=500 if FAST else 3000)
    print("(2) the same policies over the full 6 s, three definitions of success on the held-out starts")
    print(f"    {'policy':<22}{'stopped at the goal':>24}{'reached at any time':>24}{'at the goal at 6 s':>24}")
    for label, policy in [("expert", pointmass.expert)] + [(f"BC, {n} demonstrations", p) for n, p in policies.items()]:
        flags = np.array([definitions(full_rollout(sim, policy, s)) for s in held_out])
        print(f"    {label:<22}" + "".join(f"{rate(flags[:, k]):>24}" for k in range(3)))
    print("(3) the model edited in code after loading")
    path = model_path("point_mass")
    model = sim.model
    before_fp = experiment.model_fingerprint(model)
    probe = policies[5]
    before = [np.mean([sim.rollout(pol, s).success for s in held_out]) for pol in (pointmass.expert, probe)]
    model.dof_damping[:] *= 2
    after_fp = experiment.model_fingerprint(model)
    after = [np.mean([sim.rollout(pol, s).success for s in held_out]) for pol in (pointmass.expert, probe)]
    print(f"    XML file sha256 {experiment.file_sha256(path)[:16]}... before and after (the file did not change)")
    print(f"    model fingerprint {before_fp[:16]}... before, {after_fp[:16]}... after")
    print(f"    success on the held-out starts before and after doubling the damping: expert {before[0]:.2f} -> {after[0]:.2f}, "
          f"BC from 5 demonstrations {before[1]:.3f} -> {after[1]:.3f}")
    model.dof_damping[:] /= 2
    record = experiment.run_record(protocol, {"point_mass": (model, path)}, {"bc_steps": 3000, "held_out": HELD_OUT, "push_rate": 1.0}, results)
    out = experiment.save(record, "runs/l21_1_record.json")
    print(f"(0) run record written to {out}: keys {list(record)}")
    print("    " + json.dumps({k: record["software"][k] for k in ("mujoco", "numpy", "torch")}))
    print("    " + json.dumps({k: v for k, v in record["models"]["point_mass"].items() if k != "file_sha256"})[:120] + "...")

MuJoCo 3.14.0; protocol 'bc-held-out-starts' written before the runs
(1) eight training seeds, one data set: success with pushes on 50 development starts and on fresh starts
    training seed 0: development 0.58, fresh 0.595
    training seed 1: development 0.62, fresh 0.655
    training seed 2: development 0.36, fresh 0.333
    training seed 3: development 0.64, fresh 0.685
    training seed 4: development 0.50, fresh 0.480
    training seed 5: development 0.50, fresh 0.367
    training seed 6: development 0.48, fresh 0.620
    training seed 7: development 0.72, fresh 0.655
    chosen on the development set: seed 7, which scored 0.72 there and 0.655 on 400 fresh starts; the fresh scores of all seeds range from 0.333 to 0.685 (mean 0.549)
    the protocol's test: held-out interval [0.607, 0.700] against 0.6: hypothesis not falsified
(2) the same policies over the full 6 s, three definitions of success on the held-out starts
    policy                     stopped at the goal     reached at any time      at the goal at 6 s
    expert                       1.00 [0.99, 1.00]       1.00 [0.99, 1.00]       1.00 [0.99, 1.00]
    BC, 5 demonstrations         0.98 [0.96, 0.99]       0.99 [0.97, 0.99]       0.94 [0.91, 0.96]
    BC, 80 demonstrations        0.78 [0.73, 0.82]       1.00 [0.99, 1.00]       0.78 [0.73, 0.82]
(3) the model edited in code after loading
    XML file sha256 958d113c98c0fdbc... before and after (the file did not change)
    model fingerprint 9f1604a7f673cc6a... before, 89ee39add925882d... after
    success on the held-out starts before and after doubling the damping: expert 1.00 -> 1.00, BC from 5 demonstrations 0.978 -> 0.958
(0) run record written to runs/l21_1_record.json: keys ['protocol', 'software', 'models', 'config', 'results', 'written']
    {"mujoco": "3.14.0", "numpy": "2.5.3", "torch": "2.14.1+cpu"}
    {"fingerprint": "9f1604a7f673cc6a6894d86fe835c240d4ce46d52f9f83809fc01352712a6c02", "timestep": 0.01, "integrator": "mjI...

Debugging

A published number cannot be reproduced although the code is the same. The model may have been edited in code. Compare fingerprints, not file hashes.

A method looks better on the reported set than anywhere else. Something was chosen on that set: a seed, a checkpoint, a hyperparameter. Hold out a set nobody looks at until the end.

Success rates jump after a refactor. The success definition moved. Keep one function for it and test it on known trajectories.

Results depend on the training seed more than on the method. They will; report several seeds (Lesson 21.2).

Exercise

Add a fourth success definition, “within 4 cm and slower than 5 cm/s for 0.5 s”, score the three policies with it, and decide which definition matches the task you care about.

Challenge

Checkpoint 21: write a one-page protocol for a hypothesis of your own (for example, that depth input improves grasp success under a shifted camera pose), with baselines, controlled variables, seeds, splits, metric, statistical test and the falsifying result; commit it; then run it at small scale with mjcourse.experiment records and report whether it survived.

Core concept