Core concept

A learned dynamics model is trained to predict one step. A planner uses it for many: each candidate plan is rolled out by feeding the model its own predictions, so the error of step $k$ becomes part of the input of step $k+1$. That is compounding error, and it is the standard explanation for why model-based control fails. This lesson tests the explanation instead of assuming it. It trains three models with the same network and the same training, changing only the data, and measures four things separately: the one-step error, the open-loop error over 1 to 50 steps, the success of MPC as the planning horizon grows, and how far the plans the planner chooses fall short of what the model promised.

The result has two parts, and the second is the less familiar one. A modest difference in one-step error goes with the difference between solving every task and solving almost none. Yet for the good model, the planning horizon did not matter at all: errors of 2 cm at the end of a 2 s plan did not cost a single task. Compounding error is real and measurable, but whether it reaches the controller depends on how the model is used.

Visual intuition

Left: the open-loop error of the pushed puck against the prediction horizon, for the three models, on held-out episodes of the demonstrator-guided policy. All three curves rise, and the two weaker models sit two to four times above the guided one at every horizon. Right: MPC success against the planning horizon on 30 held-out tasks per point. MuJoCo as the model (hollow squares) and the guided model succeed on every task at every horizon; the other two never exceed 0.23.

Mathematics

Why errors compound

Let the true dynamics be $s_{t+1} = f(s_t, a_t)$ and the model $\hat f$, with one-step error $|\hat f(s,a) - f(s,a)| \le \varepsilon$ on the states that matter, and let $\hat f$ be Lipschitz in the state with constant $L$. The error $e_k$ of a $k$-step open-loop prediction then obeys

so $e_k \le \varepsilon\,(L^k - 1)/(L - 1)$: linear growth if $L = 1$, geometric if $L > 1$.

[!derivation] What the bound does and does not say The bound follows from the triangle inequality and the two assumptions. It assumes the one-step error holds on the predicted states $\hat s_k$, which drift away from the data the model was trained on; that is exactly where $\varepsilon$ is not controlled. The measured curve (below) grows faster than linearly beyond 10 steps, consistent with the error leaving the training distribution as it grows, but the bound alone cannot tell the two causes apart.

Three models, the same network

The three models share the MLP of Lesson 23.2 (3 hidden layers of 256, SiLU, predicting a normalized state change) and its training (20 000 steps of the one-step objective). Only the data differ:

Output (1), the one-step error of the next positions (pusher and three pucks), on held-out data of both kinds:

On random-action data, the random model is 0.22 mm worse than the guided one. Measured on guided data, all three are below 2 mm.

Output (2), the open-loop error of the pushed puck on held-out guided episodes:

For the guided model, the local slope of $\log e$ against $\log k$ is 0.8, 1.1, 1.3 and 1.4 over the four intervals of this table (computed from it): the growth is faster than linear and accelerating, as the derivation allows.

Success against the planning horizon

Output (3), the planner of Lesson 23.4 (CEM, 64 samples, 3 iterations, replanning every 0.1 s) with planning horizons of 0.4 to 2 s, 30 held-out tasks per cell:

Two facts, and they point in different directions.

First, one-step error is a poor predictor of control. The random model’s one-step error on the action distribution it was trained on is 0.77 mm against the guided model’s 0.55; its success is 0.10 against 1.00. Even on the right test distribution (guided episodes), a factor of 2.5 in one-step error (1.65 against 0.66 mm) separates every task solved from nine in ten failed. This is the objective mismatch named by Lambert et al. (arXiv:2002.04523): a model trained for one-step likelihood is evaluated on control, and the two need not agree.

Second, longer horizons did not hurt the guided model. At $H = 20$ its own prediction of the pushed puck is off by about 2 cm at the end of the plan, and it still solves all 30 tasks. Two mechanisms are at work. The planner applies only the first action of each plan and replans from the measured state (Lesson 23.4), so late-plan errors only matter through their effect on which first action wins. And candidates that share a start share most of their error: what decides the choice is the error in the differences between candidates, which is smaller than the error in any one of them (inference: consistent with the rank correlations of Lesson 23.4, not measured here as a separate quantity).

[!research] Hypothesis, falsifier, outcome Hypothesis: on this task, MPC success with a learned model is determined by its one-step error. Falsifier: two models whose one-step errors on the evaluation distribution differ by less than a factor of 3 but whose success rates differ by more than 0.5. Outcome: falsified, twice (random against guided, small against guided). A defensible replacement: success is determined by whether the model predicts how the puck responds to the pushes the planner will try, which output (5) supports and output (4) begins to measure.

Does the planner exploit the model?

A planner chooses the plan the model scores best. If the model is wrong in some direction, the search finds the plans that profit from that error, and the chosen plan does worse in reality than the model promised. Output (4) measures this at every decision of 10 learned-model episodes (horizon 12): how far the chosen plan is predicted to move the target puck toward its zone, and how far the same plan moves it in MuJoCo from the same state.

All three models over-promise on average, the signature of optimizing against a noisy score. The guided model’s plans deliver 89% of the progress it predicts. The random model’s plans are predicted to make progress and, in MuJoCo, move the puck slightly the wrong way on average; one decision in four is over-promised by more than a centimetre.

What the failures look like

Output (5), at $H = 12$:

The weak models reach the puck in 90% and 97% of episodes. They fail after that, in contact: they move the puck a tenth to a quarter of the distance the good planners do. The pusher’s own motion is easy to learn from any data (velocity commands, no contact); the puck’s response to a push is what the guided data cover and random actions rarely produce.

[!recommendation] Evaluating a world model for control Report the one-step error on the distribution the planner will visit, the open-loop error over several horizons, and closed-loop success with the simulator as reference, and keep them separate: none of them predicts the others reliably. Add one measurement of exploitation (promised against delivered progress) and one of where failures occur. A single number for model quality hides all of this.

Implementation

INPUT: tabletop.xml; the standard datasets and models of mjcourse.worldmodel.standard
PROCESS: one-step and open-loop errors of three models; MPC success at four planning horizons; promised and delivered progress; failure statistics
OUTPUT: tables; runs/l23_5/horizons.png

```python file=examples/l23_5_compounding_error.py import multiprocessing import os from concurrent.futures import ProcessPoolExecutor from pathlib import Path

import numpy as np import torch

from mjcourse import stats from mjcourse import tabletop as tt from mjcourse.worldmodel import dynamics, mpc, standard

OUT = Path(file).resolve().parents[1] / “runs” / “l23_5” FAST = os.environ.get(“MJC_FAST”) == “1” MODELS = {“guided”: “one_step”, “small”: “small_data”, “random”: “random_data”} PLAN_HORIZONS = (4, 8, 12, 20) TASKS = 6 if FAST else 30 PRED_HORIZONS = (1, 5, 10, 20, 50)

def task_and_layout(i: int): rng = np.random.default_rng([5, i]) task = (tt.OBJECTS[rng.integers(3)], list(tt.ZONES)[rng.integers(4)]) return task, tt.sample_layout(rng, task)

def control(args) -> dict: name, horizon, i = args torch.set_num_threads(1) task, layout = task_and_layout(i) cfg = mpc.PlannerConfig(horizon=horizon) if name == “MuJoCo”: factory = lambda scene: mpc.MujocoModel(scene) else: model = standard.model(MODELS[name]) factory = lambda scene: mpc.LearnedModel(model) r = mpc.run_episode(factory, task, layout, cfg=cfg) i_t = 4 + 4 * tt.OBJECTS.index(task[0]) states, zone = r[“states”], np.array(tt.ZONES[task[1]]) gap = np.linalg.norm(states[:, 0:2] - states[:, i_t:i_t + 2], axis=1).min() - tt.PUCK_RADIUS - tt.PUSHER_RADIUS progress = np.linalg.norm(states[0, i_t:i_t + 2] - zone) - np.linalg.norm(states[-1, i_t:i_t + 2] - zone) return {“model”: name, “horizon”: horizon, “i”: i, “success”: bool(r[“success”]), “touched”: bool(gap < 0.003), “progress”: float(progress)}

def exploitation(args) -> list[tuple[float, float]]: name, i = args torch.set_num_threads(1) task, layout = task_and_layout(i) scene = tt.Tabletop() scene.reset(layout) planner = mpc.Planner(mpc.LearnedModel(standard.model(MODELS[name])), task, mpc.PlannerConfig(horizon=12)) truth = mpc.MujocoModel(scene) i_t, zone = 4 + 4 * tt.OBJECTS.index(task[0]), np.array(tt.ZONES[task[1]]) pairs = [] for _ in range(tt.EPISODE_STEPS): state = scene.state() a = planner.act(state) best = planner.last[“best”] chosen = planner.last[“actions”][best:best + 1] start = np.linalg.norm(state[i_t:i_t + 2] - zone) predicted = start - np.linalg.norm(planner.last[“states”][best, -1, i_t:i_t + 2] - zone) true = start - np.linalg.norm(truth.predict(state, chosen)[0, -1, i_t:i_t + 2] - zone) pairs.append((float(predicted), float(true))) scene.step(a) if scene.success(task): break scene.close() return pairs

def one_step_error(model, episodes) -> float: s = np.concatenate([e[“states”][:-1] for e in episodes]) a = np.concatenate([e[“actions”] for e in episodes]) s2 = np.concatenate([e[“states”][1:] for e in episodes]) with torch.no_grad(): p = model(torch.as_tensor(s), torch.as_tensor(a)).numpy() idx = [0, 1, 4, 5, 8, 9, 12, 13] return float(np.linalg.norm((p - s2)[:, idx].reshape(-1, 4, 2), axis=-1).mean())

def figure(errors: dict, success: dict) -> None: import matplotlib matplotlib.use(“Agg”) import matplotlib.pyplot as plt

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(9.5, 3.8), constrained_layout=True)
colours = {"guided": "#0072B2", "small": "#E69F00", "random": "#D55E00"}
for name, err in errors.items():
    ax1.plot(PRED_HORIZONS, [1000 * err[h]["target"] for h in PRED_HORIZONS], marker="o", color=colours[name], label=name)
ax1.set_xscale("log")
ax1.set_yscale("log")
ax1.set_xlabel("prediction horizon (steps of 0.1 s)")
ax1.set_ylabel("open-loop error of the pushed puck (mm)")
ax1.grid(alpha=0.3, which="both")
ax1.legend(frameon=False)
ax1.set_title("prediction error on held-out data", fontsize=10)
for name in ("MuJoCo", "guided", "small", "random"):
    y = [success[(name, h)] for h in PLAN_HORIZONS]
    kw = {"color": "#000000", "marker": "s", "mfc": "none"} if name == "MuJoCo" else {"color": colours[name], "marker": "o"}
    ax2.plot([h * 0.1 for h in PLAN_HORIZONS], y, **kw, label=name)
ax2.set_xlabel("planning horizon (s)")
ax2.set_ylabel("MPC success rate")
ax2.set_ylim(-0.05, 1.05)
ax2.grid(alpha=0.3)
ax2.legend(frameon=False)
ax2.set_title("control success against planning horizon", fontsize=10)
fig.savefig(OUT / "horizons.png", dpi=150)
plt.close(fig)

if name == “main”: OUT.mkdir(parents=True, exist_ok=True) models = {k: standard.model(v) for k, v in MODELS.items()} guided_test, random_test = standard.dataset(“guided_test”), standard.dataset(“random_test”) print(“(1) one-step error on held-out transitions (mm)”) print(f” {‘model’:<12}{‘random test’:>16}{‘guided test’:>16}”) for name, m in models.items(): print(f” {name:<12}{1000 * one_step_error(m, random_test):>16.2f}{1000 * one_step_error(m, guided_test):>16.2f}”) pred_errors = {name: dynamics.horizon_errors(m, guided_test, PRED_HORIZONS) for name, m in models.items()} print(f”(2) open-loop error of the pushed puck against horizon (mm)”) print(f” {‘model’:<12}” + ““.join(f”{h:>9d}” for h in PRED_HORIZONS) + “ (steps of 0.1 s)”) for name, err in pred_errors.items(): print(f” {name:<12}” + ““.join(f”{1000 * err[h][‘target’]:9.1f}” for h in PRED_HORIZONS)) with ProcessPoolExecutor(min(32, os.cpu_count() or 1), mp_context=multiprocessing.get_context(“spawn”)) as pool: runs = list(pool.map(control, [(m, h, i) for m in (“MuJoCo”, “guided”, “small”, “random”) for h in PLAN_HORIZONS for i in range(TASKS)])) expl = {name: list(pool.map(exploitation, [(name, i) for i in range(TASKS)])) for name in MODELS} print(f”(3) MPC success against planning horizon, {TASKS} tasks per cell [95% Wilson interval]”) print(f” {‘model’:<12}” + ““.join(f”{f’H = {h * 0.1:.1f} s’:>22}” for h in PLAN_HORIZONS)) success_by_cell = {} for name in (“MuJoCo”, “guided”, “small”, “random”): cells = [] for h in PLAN_HORIZONS: k = sum(r[“success”] for r in runs if r[“model”] == name and r[“horizon”] == h) lo, hi = stats.wilson_interval(k, TASKS) cells.append(f”{k / TASKS:.2f} [{lo:.2f}, {hi:.2f}]”) success_by_cell[(name, h)] = k / TASKS print(f” {name:<12}” + ““.join(f”{c:>22}” for c in cells)) print(f”(4) model exploitation at planning horizon 12: predicted against true progress (m) per decision”) print(f” {‘model’:<12}{‘decisions’:>12}{‘mean predicted’:>16}{‘mean true’:>14}{‘true < predicted’:>18}”) for name in MODELS: flat = [p for ep in expl[name] for p in ep] pred, true = np.array([p[0] for p in flat]), np.array([p[1] for p in flat]) print(f” {name:<12}{len(flat):12d}{pred.mean():>16.3f}{true.mean():>14.3f}{(true < pred).mean():>18.1%}”) print(f”(5) failures at planning horizon 12: {TASKS} tasks”) print(f” {‘model’:<12}{‘never reached puck’:>20}{‘mean progress (m)’:>20}”) for name in MODELS: h12 = [r for r in runs if r[“model”] == name and r[“horizon”] == 12] print(f” {name:<12}{sum(not r[‘touched’] for r in h12):>20d}{np.mean([r[‘progress’] for r in h12]):>20.3f}”) figure(pred_errors, success_by_cell) print(f”wrote {OUT / ‘horizons.png’}”)

Output (MuJoCo 3.14.0, PyTorch 2.14 CPU, 32 worker processes; about 4 minutes):

```text
(1) one-step error of the next positions (mm), on held-out data of two kinds
  training data   random-action test   guided test
  guided                        0.55          0.66
  small                         1.15          1.58
  random                        0.77          1.65
(2) open-loop error of the pushed puck on guided test data (mm)
  training data         1        5       10       20       50   (steps of 0.1 s)
  guided              1.0      3.8      8.1     20.6     76.7
  small               2.7     15.1     30.6     60.7    131.3
  random              2.2     14.1     30.7     66.1    179.0
(3) MPC success against the planning horizon, 30 tasks per cell [95% Wilson interval]
  model                        4                   8                  12                  20
  MuJoCo       1.00 [0.89, 1.00]   1.00 [0.89, 1.00]   1.00 [0.89, 1.00]   1.00 [0.89, 1.00]
  guided       1.00 [0.89, 1.00]   1.00 [0.89, 1.00]   1.00 [0.89, 1.00]   1.00 [0.89, 1.00]
  small        0.03 [0.01, 0.17]   0.13 [0.05, 0.30]   0.07 [0.02, 0.21]   0.00 [0.00, 0.11]
  random       0.10 [0.03, 0.26]   0.23 [0.12, 0.41]   0.10 [0.03, 0.26]   0.10 [0.03, 0.26]
(4) model exploitation at planning horizon 12: progress of the target puck toward its zone over the chosen plan
  model      decisions  predicted (mm)  in MuJoCo (mm)  over-promised by > 10 mm
  guided           642            26.7            23.7                      0.08
  small           1200             4.2             2.9                      0.14
  random          1200             6.6            -0.5                      0.24
(5) the episodes at planning horizon 12: pusher reached the target puck, and the puck's progress toward its zone
  model       success  reached the puck  mean progress (mm)
  MuJoCo         1.00              1.00               225.9
  guided         1.00              1.00               223.2
  small          0.07              0.90                25.6
  random         0.10              0.97                52.6

Debugging

An exploitation measure that shows nothing. The first version of output (4) compared the predicted and true cost of the chosen plan and found ratios of 1.00 to 1.01 for every model, including the random one. The cost is dominated by the puck’s distance to its zone, which hardly changes within 1.2 s, so any model scores it well. The progress of the puck over the plan is the part of the prediction the choice depends on; measured that way, the difference between models appears.

Comparing one-step errors on the wrong data. On random-action test data the random model looks almost as good as the guided one (0.77 against 0.55 mm). Always evaluate a model for planning on the state and action distribution the planner will produce; here, the guided test set is the closer proxy.

A flat success curve read as “horizon does not matter”. It does not matter for this task, this cost (which tells the planner where to push from) and this replanning rate. With a cost that rewards only the final puck position, short horizons would see no reward at all; with open-loop execution, late-plan errors are acted on. Lesson 23.6 removes the replanning and measures the difference.

Exercise

Train the guided model with the multi-step objective (dynamics.fit(..., objective="multi_step", k=5), which backpropagates through 5-step rollouts) and repeat outputs (1) to (3). Predict, before running: does the one-step error rise, fall or stay? Does the 20-step error? Does control change?

Challenge

Make horizon matter. Remove the pusher-placement term from the cost, so the planner is rewarded only for the puck’s distance to its zone, and repeat output (3) for MuJoCo and the guided model. State the planning horizon below which MuJoCo itself fails, and whether the learned model fails earlier.

Research connection

[!research] Compounding error in model-based learning Talvitie (arXiv:1612.06018) shows that a model’s ability to correct its own errors is more tightly related to model-based performance than its one-step error, and trains models on their own predictions to that end; the multi-step objective of the exercise is in the same family. MBPO (Janner et al., arXiv:1906.08253) keeps model rollouts short and branched from real states for the same reason: long imagined rollouts accumulate error. PETS (Chua et al., arXiv:1805.12114) plans with an ensemble and propagates its uncertainty, one answer to exploitation. The experiments here are on one task with one network family; how far their conclusions transfer to latent models of images (Lesson 23.3) is an open question for the reader’s own project (Lesson 23.7).

{"id": "23.5-check", "title": "Knowledge check", "questions": [
  {"kind": "mcq", "q": "The random-data model's one-step error on random-action data is 0.77 mm, the guided model's 0.55 mm. Their MPC success rates?",
   "options": ["About equal", "0.10 against 1.00", "0.50 against 0.70", "The random model is better"],
   "answer": 1,
   "explain": "<p>A small one-step difference hides a large control difference: objective mismatch.</p>"},
  {"kind": "mcq", "q": "Why did a planning horizon of 2 s not hurt the guided model despite about 2 cm of error at the end of each plan?",
   "options": ["The error was zero", "Only the first action is applied before replanning, and the choice depends on differences between candidates more than on any one prediction", "Longer horizons always help", "MuJoCo corrected the model"],
   "answer": 1,
   "explain": "<p>Replanning and shared error between candidates keep late-plan error away from the decision.</p>"},
  {"kind": "mcq", "q": "Where do the weak models fail?",
   "options": ["They never reach the puck", "They reach it in 90 to 97% of episodes but barely move it toward the zone", "They move the wrong puck", "They time out before planning"],
   "answer": 1,
   "explain": "<p>The pusher's free motion is easy to learn; the puck's response to pushes is not covered by their data.</p>"},
  {"kind": "mcq", "q": "What does a positive gap between predicted and delivered progress of the chosen plan indicate?",
   "options": ["A bug in MuJoCo", "The planner is selecting plans that profit from the model's errors (exploitation)", "The task is solved", "The horizon is too short"],
   "answer": 1,
   "explain": "<p>Optimizing against an imperfect score selects for its errors.</p>"}
]}

Next

Lesson 23.6 changes the world under the model (friction, mass, size, puck positions, impact speeds) and measures prediction and control separately, with and without replanning.