A success rate is a measurement with two sources of noise. One is the evaluation episodes: which starts were drawn, and which pushes came. The other is training: the seed, the data order, the initialization. Intervals and tests that see only the first are wrong whenever the second is large, and in learning it usually is. This lesson plans an evaluation from a target interval width, then uses real data, two behaviour-cloning methods each trained with 8 seeds and evaluated on the same 400 starts, to show what pooled intervals, pooled tests, paired designs and aggregation across tasks do with that data. Lesson 13.3 introduced the statistics; this lesson is about choosing them.
[!established] Planning episodes from the interval you need For a success rate $p$ estimated from $n$ independent episodes, a 95% interval has half-width about $1.96\sqrt{p(1-p)/n}$, so $n \approx 1.96^2\, p(1-p)/h^2$ (
mjcourse.stats.episodes_for_halfwidth):
$h$ = 0.10 $h$ = 0.05 $h$ = 0.02 $p$ = 0.5 97 385 2401 $p$ = 0.8 62 246 1537 $p$ = 0.95 19 73 457 These are episodes per trained policy. They say nothing about how many training seeds are needed; that is a separate question, and below it decides the answer.
There is no lab for this lesson. Picture the 16 trained policies as 16 rows of 400 outcomes: the rows differ from each other far more than outcomes within a row differ from chance.
The point-mass task with random pushes (Level 14), behaviour cloning from 20 and from 80 clean demonstrations, 8 training seeds each, every policy evaluated on the same 400 starts with the same pushes:
| Method | Success per training seed | Pooled over 3200 episodes, Wilson | Over the 8 seeds, bootstrap |
|---|---|---|---|
| BC, 20 demonstrations | 0.588 0.690 0.328 0.695 0.515 0.383 0.595 0.650 | 0.555 [0.538, 0.572] | 0.555 [0.462, 0.638] |
| BC, 80 demonstrations | 0.407 0.605 0.547 0.695 0.515 0.367 0.585 0.698 | 0.552 [0.535, 0.570] | 0.552 [0.475, 0.626] |
The same code and data give 0.33 to 0.70 depending on the seed. The pooled interval treats 3200 episodes as independent draws from one policy and is five times narrower than the interval over seeds. Only the second describes the method; the first describes these eight policies, as if they were one.
The difference, 20 minus 80 demonstrations, is +0.003. Over seeds its bootstrap interval is [−0.117, +0.120]: these data cannot tell the methods apart, and could not exclude a difference of 0.1 either way. Paired by start, averaging each start’s outcome over the seeds and differencing, the interval is [−0.016, +0.021]. Pairing removes the variation between starts, which is plausibly large for this task since some starts are easy for every policy (inference), but it answers a narrower question: whether these 16 trained policies differ on these 400 starts, not whether the methods do. Use pairing for the start-level variation and seeds for the method-level claim; report the latter for a method comparison.
Drawing small studies from the same data shows what a typical paper-sized experiment would report. Each study uses $k$ seeds per method and 100 episodes per seed, and is repeated 2000 times:
| Seeds per method | 1 | 2 | 3 | 5 |
|---|---|---|---|---|
| studies in which the better method wins | 0.50 | 0.50 | 0.52 | 0.50 |
| studies in which a pooled Fisher test says “significant” (p < 0.05) | 0.44 | 0.40 | 0.37 | 0.26 |
Over all the data the methods differ by only 0.003, so the winner of a small study is essentially a coin flip. A Fisher exact test on pooled episodes, which assumes every episode independent, still calls the difference significant in 26 to 44% of studies, five to nine times its nominal 5% false-positive rate. More seeds bring it down, but even five per method leave it far from 5%. A test must use the training seed as its unit, as the seed bootstrap above does, or model both levels.
One policy (BC, 20 demonstrations, seed 0) on two tasks: 400 clean starts and 100 pushed starts. It succeeds on all 400 clean starts (1.000) and on 0.580 of the pushed ones. The mean over episodes is 0.916; the mean over tasks is 0.790. The per-episode mean weighs tasks by how many episodes they happened to get, so it rewards evaluating easy tasks more. Benchmarks with tasks of different sizes need the per-task mean, or both numbers with the episode counts.
[!recommendation] A methodology plan, before running Fix the episodes per policy from the interval you need (the table above). Train every method with several seeds, at least 5 here and more when methods are close. Evaluate every policy on the same starts and disturbances, so that comparisons can be paired. Report the interval over seeds; use pairing to narrow comparisons only within a seed-level analysis. Aggregate tasks per task. Change one factor per ablation and keep everything else, including seeds and starts, fixed.
INPUT: the point-mass task with pushes; BC from 20 and from 80 demonstrations, 8 training seeds each; the same 400 starts and pushes for every policy
PROCESS: episode planning; per-seed success, pooled and seed-level intervals; the difference over seeds and paired by start; small studies resampled from the data; per-episode and per-task aggregation
OUTPUT: printed tables
```python file=examples/l21_2_methodology.py “"”Lesson 21.2: seeds, episodes, intervals and comparisons, on real evaluation data (Level 21).
INPUT the point-mass task of Level 14 with random pushes (1 per second); two methods: behaviour cloning from 20 and from 80 clean demonstrations; each trained with 8 seeds; every policy evaluated on the same 400 held-out starts with the same pushes (a paired design) PROCESS (1) planning: episodes for a target interval half-width (mjcourse.stats.episodes_for_halfwidth); (2) each method’s success: per seed, a Wilson interval pooled over all 3200 episodes, and a bootstrap interval over the 8 seeds; (3) the difference between the methods: over seeds, and paired by start (each start’s success averaged over seeds, then differenced); (4) small studies drawn from these data: with k seeds per method and 100 episodes per seed, how often the better method wins, and how often a pooled Fisher test calls the difference significant; (5) aggregating tasks of unequal size: the clean task (400 starts) and the pushed task (100 starts), per episode and per task OUTPUT printed tables
Run: python examples/l21_2_methodology.py (about 8 minutes) “””
import os
import numpy as np from scipy.stats import fisher_exact
from mjcourse.il import bc, pointmass from mjcourse.stats import bootstrap_ci, episodes_for_halfwidth, per_episode_mean, per_task_mean, wilson_interval
FAST = os.environ.get(“MJC_FAST”) == “1” SEEDS = 2 if FAST else 8 N_EVAL = 60 if FAST else 400
def train(n_demos: int, seed: int): sim = pointmass.PointMass() starts = pointmass.sample_starts(np.random.default_rng(1), n_demos) runs = [sim.rollout(pointmass.expert, s) for s in starts] policy, _ = bc.fit(np.vstack([r.states for r in runs]), np.vstack([r.actions for r in runs]), steps=500 if FAST else 3000, seed=seed) return policy
def outcomes(policy, starts: np.ndarray, push_rate: float) -> np.ndarray: “"”Per-start success (0/1); the i-th start’s pushes come from a fixed generator, shared by all policies.””” return np.array([r.success for r in pointmass.evaluate(policy, starts, push_rate=push_rate, seed=12)[“runs”]], float)
if name == “main”: print(“(1) episodes needed for a 95% interval of half-width h at success rate p”) print(f” {‘’:<8}” + ““.join(f”{‘h = ‘ + str(h):>10}” for h in (0.10, 0.05, 0.02))) for p in (0.5, 0.8, 0.95): print(f” {‘p = ‘ + str(p):<8}” + ““.join(f”{episodes_for_halfwidth(h, p):>10}” for h in (0.10, 0.05, 0.02))) starts = pointmass.sample_starts(np.random.default_rng(4), N_EVAL) data = {} for n in (20, 80): data[n] = np.array([outcomes(train(n, seed), starts, 1.0) for seed in range(SEEDS)]) # (seeds, starts) print(f”(2) success with pushes, {SEEDS} training seeds x {N_EVAL} starts per method”) for n, x in data.items(): per_seed = x.mean(axis=1) lo, hi = wilson_interval(int(x.sum()), x.size) est, blo, bhi = bootstrap_ci(per_seed) print(f” BC from {n} demonstrations: per seed “ + “ “.join(f”{v:.3f}” for v in per_seed)) print(f” pooled over {x.size} episodes: {x.mean():.3f}, Wilson [{lo:.3f}, {hi:.3f}]; “ f”over seeds: {est:.3f}, bootstrap [{blo:.3f}, {bhi:.3f}]”) print(“(3) the difference, BC from 20 minus BC from 80”) a, b = data[20].mean(axis=1), data[80].mean(axis=1) rng = np.random.default_rng(0) boot = [rng.choice(a, a.size).mean() - rng.choice(b, b.size).mean() for _ in range(10000)] print(f” over seeds: {a.mean() - b.mean():+.3f}, bootstrap [{np.percentile(boot, 2.5):+.3f}, {np.percentile(boot, 97.5):+.3f}]”) per_start = data[20].mean(axis=0) - data[80].mean(axis=0) # paired by start boot = [rng.choice(per_start, per_start.size).mean() for _ in range(10000)] print(f” paired by start (start-level variation only; seeds averaged): {per_start.mean():+.3f}, “ f”bootstrap [{np.percentile(boot, 2.5):+.3f}, {np.percentile(boot, 97.5):+.3f}]”) print(“(4) small studies drawn from these data: k seeds per method, 100 episodes per seed, 2000 studies each”) better = 20 if a.mean() > b.mean() else 80 print(f” (the better method over all data: BC from {better} demonstrations)”) for k in ((1, 2) if FAST else (1, 2, 3, 5)): wins = significant = 0 for _ in range(2000): cols = rng.choice(N_EVAL, min(100, N_EVAL), replace=False) xa = data[20][rng.choice(SEEDS, k, replace=False)][:, cols] xb = data[80][rng.choice(SEEDS, k, replace=False)][:, cols] wins += (xa.mean() > xb.mean()) == (better == 20) table = [[xa.sum(), xa.size - xa.sum()], [xb.sum(), xb.size - xb.sum()]] significant += fisher_exact(table)[1] < 0.05 print(f” k={k}: the better method wins in {wins / 2000:.2f} of studies; a pooled Fisher test calls the “ f”difference significant in {significant / 2000:.2f}”) print(“(5) two tasks of unequal size, BC from 20 demonstrations (seed 0): clean 400 starts, pushed 100 starts”) policy = train(20, 0) clean = outcomes(policy, starts, 0.0) pushed = outcomes(policy, starts[:min(100, N_EVAL)], 1.0) successes = np.r_[clean, pushed] tasks = np.r_[np.zeros(clean.size), np.ones(pushed.size)] print(f” clean {clean.mean():.3f}, pushed {pushed.mean():.3f}; mean per episode {per_episode_mean(successes):.3f}, “ f”mean per task {per_task_mean(successes, tasks):.3f}”)
Output:
```text
(1) episodes needed for a 95% interval of half-width h at success rate p
h = 0.1 h = 0.05 h = 0.02
p = 0.5 97 385 2401
p = 0.8 62 246 1537
p = 0.95 19 73 457
(2) success with pushes, 8 training seeds x 400 starts per method
BC from 20 demonstrations: per seed 0.588 0.690 0.328 0.695 0.515 0.383 0.595 0.650
pooled over 3200 episodes: 0.555, Wilson [0.538, 0.572]; over seeds: 0.555, bootstrap [0.462, 0.638]
BC from 80 demonstrations: per seed 0.407 0.605 0.547 0.695 0.515 0.367 0.585 0.698
pooled over 3200 episodes: 0.552, Wilson [0.535, 0.570]; over seeds: 0.552, bootstrap [0.475, 0.626]
(3) the difference, BC from 20 minus BC from 80
over seeds: +0.003, bootstrap [-0.117, +0.120]
paired by start (start-level variation only; seeds averaged): +0.003, bootstrap [-0.016, +0.021]
(4) small studies drawn from these data: k seeds per method, 100 episodes per seed, 2000 studies each
(the better method over all data: BC from 20 demonstrations)
k=1: the better method wins in 0.50 of studies; a pooled Fisher test calls the difference significant in 0.44
k=2: the better method wins in 0.50 of studies; a pooled Fisher test calls the difference significant in 0.40
k=3: the better method wins in 0.52 of studies; a pooled Fisher test calls the difference significant in 0.37
k=5: the better method wins in 0.50 of studies; a pooled Fisher test calls the difference significant in 0.26
(5) two tasks of unequal size, BC from 20 demonstrations (seed 0): clean 400 starts, pushed 100 starts
clean 1.000, pushed 0.580; mean per episode 0.916, mean per task 0.790
Intervals are tight and the next run disagrees. The interval counted episodes and ignored training seeds. Recompute over seeds.
A significant improvement does not replicate. Check the unit of the test: episodes pooled across seeds overstate the evidence, as in the small-study table.
Two methods trade places across papers. Their difference is smaller than the seed-to-seed spread. Report that spread.
An aggregate score rewards evaluating easy tasks more. It is a per-episode mean over tasks of unequal size. Use the per-task mean.
From the same data, find the number of seeds per method needed for the seed bootstrap interval of the difference to have a half-width of 0.05, by resampling seeds with replacement. Compare with the episodes table: which costs more, seeds or episodes?
Design and run an ablation with two factors, the number of demonstrations (20, 80) and pushes in the demonstrations (none, one per second), with 5 seeds per cell on shared starts. Estimate both main effects and their interaction with seed-level intervals, and state which effects the data can and cannot support.
[!research] Statistics in learning papers Henderson et al. (arXiv:1709.06560) and Agarwal et al. (arXiv:2108.13264), both in the reading list, document how seed variation and small numbers of runs mislead deep RL comparisons, and Agarwal et al. recommend interval estimates over runs (with the interquartile mean,
mjcourse.stats.iqm). The small-study table above is the same lesson measured on a MuJoCo imitation-learning task: the unit of replication is the training run, not the episode.
{"id": "21.2-check", "title": "Knowledge check", "questions": [
{"kind": "numeric", "q": "How many episodes give a 95% interval of half-width 0.05 at a success rate of 0.5?",
"answer": 385, "tol": 2,
"explain": "<p>1.96^2 x 0.25 / 0.05^2 = 384.2, rounded up.</p>"},
{"kind": "mcq", "q": "BC from 20 demonstrations: pooled Wilson [0.538, 0.572], over seeds [0.462, 0.638]. Which describes the method?",
"options": ["The pooled one: more episodes", "The one over seeds: the method's result varies with the training seed", "Both equally", "Neither"],
"answer": 1,
"explain": "<p>Per-seed success ranges from 0.33 to 0.70; the pooled interval ignores that.</p>"},
{"kind": "mcq", "q": "Two equal methods, one seed each, 100 episodes each. How often did a pooled Fisher test call the difference significant?",
"options": ["About 5%", "About 44%", "Never", "Always"],
"answer": 1,
"explain": "<p>Seed variation is not in the test's model, so it is read as a method difference.</p>"},
{"kind": "mcq", "q": "A clean task with 400 episodes at 1.000 and a pushed task with 100 at 0.580. Which average treats the tasks equally?",
"options": ["0.916, per episode", "0.790, per task"],
"answer": 1,
"explain": "<p>The per-episode mean weighs the clean task four times as much.</p>"}
]}
Lesson 21.3 reads benchmarks along the same lines: what is fixed, what varies, and what a number means.