Behaviour cloning turns demonstrations into a policy by regression: record the states an expert visited and the actions it took, and fit a function from one to the other. No reward is needed, which is why cloning is the default way to learn manipulation from human demonstrations. Its weakness is structural. The regression is fit on the expert’s state distribution, but the clone is deployed on its own: every small error takes it to states the expert never visited, where its predictions are not supported by data, and errors there take it further away. This is covariate shift, and it is why a clone can match the expert’s actions closely on held-out demonstration states and still fail in closed loop.
DAgger (Dataset Aggregation) attacks the cause: let the clone drive, ask the expert what it would have done in every state the clone visits, add those labels to the dataset and fit again. After a few iterations the data covers the states the clone actually reaches. This lesson measures both on the point-mass task, together with a third remedy that teaches the same thing from a different side.
[!established] DAgger Ross, Gordon and Bagnell (arXiv:1011.0686) showed that behaviour cloning’s error can grow quadratically with the task horizon, because mistakes move the learner off the expert’s state distribution, and that iteratively aggregating expert labels on the learner’s own state distribution reduces this to a linear dependence. The algorithm in
mjcourse.il.bc.daggeris theirs, with the mixing coefficient set to zero after the first iteration.
Record 5 demonstrations, then Test on 20 starts. The learner here is a nearest-neighbour lookup in the recorded data, the simplest possible clone. Tests run with random pushes (1 per second by default), which the demonstrations did not contain.
The nearest-neighbour clone is a much weaker learner than the script’s MLP (1 of 20 against 53 of 100 after 20 clean demonstrations), because it cannot interpolate between demonstrated states at all. The pattern across the three remedies is the same.
```lab bc {“dock”: true, “title”: “Cloning an expert, and DAgger”, “height”: 340}
## Mathematics
### The task and the expert
The state is $s = (x, y, \dot x, \dot y)$ of a 0.5 kg mass, the action a force in $[-1, 1]^2$ N applied every 10 ms. Starts are drawn left of a round obstacle (radius 0.1 m) and the goal is on its right; success is stopping within 4 cm of it, slower than 5 cm/s, within 6 s. The expert steers with a saturated PD law to a waypoint 0.2 m above or below the obstacle, on the side the mass is already on, until it has passed it, then to the goal. Output (1): without pushes it succeeds from all 100 held-out starts in 2.36 s on average; with pushes (on average one per second, a velocity change of 0.4 m/s in a random direction) it succeeds in 95 of 100.
### Cloning, and where it fails
A two-layer MLP (64 tanh units) is fit by mean-squared error to 20 demonstrations recorded without pushes (4 700 state-action pairs). Output (2): without pushes it succeeds in 100 of 100 episodes; with pushes, in 53, against the expert's 95. The pushes are the same for both, so the gap is the clone's.
The cause is measurable. For every state visited during the pushed episodes, output (2) computes the distance to the nearest demonstrated state, in units of the demonstrations' standard deviation:
| Policy under pushes | Median | 90th percentile | 99th percentile | Furthest point: successes / failures |
|---|---|---|---|---|
| expert | 0.38 | 1.26 | 2.79 | 2.47 / 3.65 |
| clone | 1.08 | 2.75 | 4.28 | 2.77 / 4.07 |
The expert, pushed, also leaves the demonstrated states, but its feedback law does not need data to recover. The clone spends its episodes three times further from the data, and the episodes it fails are those that went furthest.
### DAgger, more data, and data with recoveries
Output (3) compares three ways to spend expert labels, on 3 seeds, with success measured under pushes on the same 100 held-out starts:
- **DAgger:** start from the 20 clean demonstrations; each iteration runs the current clone from 4 new starts, with pushes, and the expert labels every visited state. Five iterations.
- **More demonstrations:** 20, 30, 40 and 50 clean demonstrations.
- **Demonstrations recorded with pushes:** the same numbers of demonstrations, recorded while the expert was being pushed.
*Figure: one line per seed. Regenerate with `examples/l14_1_bc_dagger.py` (runs/l14_1/labels.png).*
At about 11 000 to 12 000 labels, DAgger reaches 0.92, 0.95 and 0.94; more clean demonstrations reach 0.52, 0.63 and 0.54, no better than 20 of them; demonstrations recorded with pushes reach 0.92 to 0.95 at every budget tried. A single DAgger iteration (1 300 to 2 200 new labels) already lifts success from 0.53 to 0.64 to between 0.85 and 0.99. Quantity of data is not the issue; coverage is. DAgger obtains it by querying the expert where the learner goes, pushed demonstrations obtain it by perturbing the expert where it goes, and both work here because a push is the kind of error the clone makes. The pushed-demonstration remedy needs to know which perturbations matter; DAgger does not, but it needs an expert that can be queried in arbitrary states, which a human teleoperator often cannot do.
### Regression averages what it cannot separate
The expert's lateral force is discontinuous at $y = 0$: at $x = -0.2$ m and at rest it is −0.79 N just below the axis and +0.80 N just above. Output (4) queries the clone across $y = 0$: it outputs +0.04 to +0.29 N over the whole range from −3 to +3 cm. A mean-squared-error fit of a smooth function to two branches settles between them. Here it leans toward one side and pushes only weakly, which is why the clone touches the obstacle in 6% of episodes even without pushes, where the expert never does. More data near $y = 0$ sharpens the transition but cannot make a smooth regressor choose; policies that represent several modes (mixture models, diffusion policies) can.
## Implementation
```io
INPUT: point_mass.xml; the scripted expert; mjcourse.il.bc
PROCESS: evaluate the expert and a clone with and without pushes; measure distances to the demonstrations; compare DAgger, more demonstrations and pushed demonstrations at equal label budgets on 3 seeds; query the clone across the expert's discontinuity
OUTPUT: printed tables and a figure
```python file=src/mjcourse/il/pointmass.py “"”The point-mass navigation task and its scripted expert (Level 14).
State s = (x, y, vx, vy); action a = (fx, fy) in [-1, 1] N, applied every 10 ms physics step. The mass starts left of a round obstacle and must stop at the goal on its right: within 4 cm of it, slower than 5 cm/s. An episode lasts at most 6 s.
With push_rate > 0, the mass is kicked at random times (a velocity change of push
m/s in a random direction, on average push_rate times per second): the states that
follow are ones a demonstration without kicks never shows.
The expert goes around the obstacle on the side it is already on: it steers to a waypoint 0.2 m above or below the obstacle’s centre until it has passed it, then to the goal, with a saturated PD law. Its action is therefore discontinuous at y = 0, where the side flips. “””
from future import annotations
from dataclasses import dataclass
import mujoco import numpy as np
from mjcourse.paths import model_path
GOAL = np.array([0.35, 0.0]) START_LOW, START_HIGH = np.array([-0.40, -0.25]), np.array([-0.30, 0.25]) MAX_STEPS = 600
def expert(state: np.ndarray) -> np.ndarray: p, v = state[:2], state[2:] side = 1.0 if p[1] >= 0 else -1.0 target = np.array([0.0, 0.2 * side]) if p[0] < -0.02 else GOAL return np.clip(4.0 * (target - p) - 1.5 * v, -1.0, 1.0)
def sample_starts(rng: np.random.Generator, n: int) -> np.ndarray: return rng.uniform(START_LOW, START_HIGH, (n, 2))
@dataclass class Rollout: states: np.ndarray # (T, 4) states at which actions were chosen actions: np.ndarray # (T, 2) actions taken success: bool hit_obstacle: bool steps: int
class PointMass: def init(self): self.model = mujoco.MjModel.from_xml_path(str(model_path(“point_mass”))) self.data = mujoco.MjData(self.model) self.obstacle = self.model.geom(“obstacle”).id
def rollout(self, policy, start: np.ndarray, max_steps: int = MAX_STEPS, push_rate: float = 0.0,
push: float = 0.4, rng: np.random.Generator | None = None) -> Rollout:
m, d = self.model, self.data
mujoco.mj_resetData(m, d)
d.qpos[:] = start
mujoco.mj_forward(m, d)
states, actions, hit = [], [], False
p_push = push_rate * m.opt.timestep
for t in range(max_steps):
if p_push > 0 and rng.random() < p_push:
angle = rng.uniform(0, 2 * np.pi)
d.qvel[:] += push * np.array([np.cos(angle), np.sin(angle)])
s = np.r_[d.qpos, d.qvel]
a = np.clip(np.asarray(policy(s), dtype=float), -1.0, 1.0)
states.append(s)
actions.append(a)
d.ctrl[:] = a
mujoco.mj_step(m, d)
hit |= any(self.obstacle in (c.geom1, c.geom2) for c in d.contact[:d.ncon])
if np.linalg.norm(d.qpos - GOAL) < 0.04 and np.linalg.norm(d.qvel) < 0.05:
return Rollout(np.array(states), np.array(actions), True, hit, t + 1)
return Rollout(np.array(states), np.array(actions), False, hit, max_steps)
def evaluate(policy, starts: np.ndarray, push_rate: float = 0.0, seed: int = 0) -> dict: “"”Closed-loop success, obstacle contacts and time on the given starts; with pushes, the i-th episode’s kicks come from a generator seeded with (seed, i), so policies are compared on the same kicks.””” sim = PointMass() runs = [sim.rollout(policy, s, push_rate=push_rate, rng=np.random.default_rng([seed, i])) for i, s in enumerate(starts)] return {“success”: float(np.mean([r.success for r in runs])), “hit”: float(np.mean([r.hit_obstacle for r in runs])), “steps”: float(np.mean([r.steps for r in runs if r.success])) if any(r.success for r in runs) else float(“nan”), “runs”: runs}
```python file=src/mjcourse/il/bc.py
"""Behaviour cloning by regression, and DAgger (Level 14).
`fit` trains an MLP that maps states to actions by mean-squared error, with inputs and
outputs standardized by the dataset's statistics. `dagger` alternates: run the current
policy, ask the expert what it would have done in every state visited, add those labels
to the dataset, and fit again (Ross, Gordon and Bagnell, arXiv:1011.0686, with beta = 0
after the first iteration).
"""
from __future__ import annotations
import numpy as np
import torch
from torch import nn
class Policy:
"""A trained regressor with its normalization; callable on one state."""
def __init__(self, net: nn.Module, x_mean, x_std, y_mean, y_std):
self.net, self.x_mean, self.x_std, self.y_mean, self.y_std = net, x_mean, x_std, y_mean, y_std
@torch.no_grad()
def __call__(self, state: np.ndarray) -> np.ndarray:
x = torch.as_tensor((np.asarray(state) - self.x_mean) / self.x_std, dtype=torch.float32)
return self.net(x).numpy() * self.y_std + self.y_mean
def fit(states: np.ndarray, actions: np.ndarray, steps: int = 3000, hidden: int = 64, lr: float = 1e-3,
batch: int = 256, seed: int = 0, validation: tuple[np.ndarray, np.ndarray] | None = None) -> tuple[Policy, dict]:
"""Returns the policy and the training (and validation) mean-squared error in normalized units."""
torch.manual_seed(seed)
x_mean, x_std = states.mean(axis=0), states.std(axis=0) + 1e-6
y_mean, y_std = actions.mean(axis=0), actions.std(axis=0) + 1e-6
x = torch.as_tensor((states - x_mean) / x_std, dtype=torch.float32)
y = torch.as_tensor((actions - y_mean) / y_std, dtype=torch.float32)
net = nn.Sequential(nn.Linear(states.shape[1], hidden), nn.Tanh(), nn.Linear(hidden, hidden), nn.Tanh(),
nn.Linear(hidden, actions.shape[1]))
optimizer = torch.optim.Adam(net.parameters(), lr=lr)
gen = torch.Generator().manual_seed(seed)
for _ in range(steps):
idx = torch.randint(0, len(x), (min(batch, len(x)),), generator=gen)
loss = (net(x[idx]) - y[idx]).pow(2).mean()
optimizer.zero_grad()
loss.backward()
optimizer.step()
with torch.no_grad():
info = {"train_mse": float((net(x) - y).pow(2).mean())}
if validation is not None:
vx = torch.as_tensor((validation[0] - x_mean) / x_std, dtype=torch.float32)
vy = torch.as_tensor((validation[1] - y_mean) / y_std, dtype=torch.float32)
info["val_mse"] = float((net(vx) - vy).pow(2).mean())
return Policy(net, x_mean, x_std, y_mean, y_std), info
def dagger(sim, expert, states: np.ndarray, actions: np.ndarray, starts_per_iteration: np.ndarray,
iterations: int, evaluate_fn, seed: int = 0, push_rate: float = 0.0) -> list[dict]:
"""DAgger from an initial dataset. Each iteration rolls out the current policy from the
next block of starts, labels every visited state with the expert, aggregates and refits.
Returns one row per policy (iteration 0 is plain behaviour cloning)."""
rows = []
for it in range(iterations + 1):
policy, _ = fit(states, actions, seed=seed)
rows.append({"iteration": it, "labels": len(states), **evaluate_fn(policy)})
if it == iterations:
break
for j, start in enumerate(starts_per_iteration[it]):
visited = sim.rollout(policy, start, push_rate=push_rate, rng=np.random.default_rng([seed, 1000 + it, j])).states
states = np.vstack([states, visited])
actions = np.vstack([actions, np.array([expert(s) for s in visited])])
return rows
```python file=examples/l14_1_bc_dagger.py “"”Lesson 14.1: behaviour cloning, covariate shift and DAgger on the point-mass task (Level 14 checkpoint).
INPUT mjcourse.il.pointmass (task and scripted expert), mjcourse.il.bc (cloning and DAgger) PROCESS (1) the expert on 100 held-out starts, without and with random pushes (1 per second on average, 0.4 m/s each); (2) behaviour cloning on 20 expert demonstrations recorded without pushes, evaluated without and with pushes; covariate shift measured as the distance from every visited state to the nearest demonstrated state, for the expert and the clone; (3) success under pushes against the number of expert labels, 3 seeds: DAgger (5 iterations of 4 rollouts), cloning on more demonstrations without pushes, and cloning on demonstrations recorded with pushes; a figure; (4) the expert’s discontinuity at y = 0 and what regression makes of it OUTPUT printed tables and runs/l14_1/labels.png
Run: python examples/l14_1_bc_dagger.py “””
import os from multiprocessing import Pool from pathlib import Path
import numpy as np import torch
from mjcourse.il import bc from mjcourse.il import pointmass as pm
FAST = os.environ.get(“MJC_FAST”) == “1” PUSH = 1.0 # pushes per second at evaluation HELD_OUT = pm.sample_starts(np.random.default_rng(999), 30 if FAST else 100) FIGURE = Path(file).resolve().parents[1] / “runs” / “l14_1” / “labels.png”
def demonstrations(n: int, seed: int, push_rate: float = 0.0) -> tuple[np.ndarray, np.ndarray]: sim, rng = pm.PointMass(), np.random.default_rng(seed) runs = [sim.rollout(pm.expert, s, push_rate=push_rate, rng=np.random.default_rng([seed, 7, i])) for i, s in enumerate(pm.sample_starts(rng, n))] return np.vstack([r.states for r in runs]), np.vstack([r.actions for r in runs])
def scores(policy, push_rate: float = PUSH) -> dict: out = pm.evaluate(policy, HELD_OUT, push_rate=push_rate) return {k: out[k] for k in (“success”, “hit”, “steps”)} | {“runs”: out[“runs”]}
def nearest_distance(points: np.ndarray, data: np.ndarray, scale: np.ndarray) -> np.ndarray:
“"”Distance from each point to its nearest neighbour in data, in units of the data’s std.”””
a, b = points / scale, data / scale
out = np.empty(len(a))
for i in range(0, len(a), 2000):
out[i:i + 2000] = np.sqrt(((a[i:i + 2000, None, :] - b[None, :, :]) ** 2).sum(-1)).min(axis=1)
return out
def part1_2() -> None: torch.set_num_threads(4) for label, rate in ((“without pushes”, 0.0), (“with pushes”, PUSH)): s = scores(pm.expert, rate) print(f” expert {label:<16} success {s[‘success’]:.2f}, obstacle touched {s[‘hit’]:.2f}, mean time {s[‘steps’] / 100:.2f} s”) states, actions = demonstrations(20, seed=0) policy, info = bc.fit(states, actions, seed=0) print(f” 20 demonstrations: {len(states)} state-action pairs; training MSE {info[‘train_mse’]:.4f} (normalized units)”) for label, rate in ((“without pushes”, 0.0), (“with pushes”, PUSH)): s = scores(policy, rate) print(f” clone {label:<16} success {s[‘success’]:.2f}, obstacle touched {s[‘hit’]:.2f}, mean time {s[‘steps’] / 100:.2f} s”) scale = states.std(axis=0) print(f” distance from visited states to the nearest demonstrated state (units of the demonstrations’ std), with pushes:”) for name, pol in ((“expert”, pm.expert), (“clone”, policy)): runs = scores(pol)[“runs”] visited = np.vstack([r.states for r in runs]) dist = nearest_distance(visited, states, scale) per_run = np.array([nearest_distance(r.states, states, scale).max() for r in runs]) failed = np.array([not r.success for r in runs]) line = f” {name:<7} median {np.median(dist):.2f}, 90th percentile {np.percentile(dist, 90):.2f}, 99th {np.percentile(dist, 99):.2f}” if failed.any() and (~failed).any(): line += (f”; largest distance per episode: successes median {np.median(per_run[~failed]):.2f}, “ f”failures median {np.median(per_run[failed]):.2f}”) print(line)
def curve(seed: int) -> dict: torch.set_num_threads(1) rng = np.random.default_rng([seed, 1]) states, actions = demonstrations(20, seed=seed) sim = pm.PointMass() evaluate = lambda p: {k: v for k, v in scores(p).items() if k != “runs”} # noqa: E731 iterations = 2 if FAST else 5 dagger_rows = bc.dagger(sim, pm.expert, states, actions, pm.sample_starts(rng, iterations * 4).reshape(iterations, 4, 2), iterations, evaluate, seed=seed, push_rate=PUSH) more, pushed = [], [] for n in ((20, 40) if FAST else (20, 30, 40, 50)): for rows, rate in ((more, 0.0), (pushed, PUSH)): s, a = demonstrations(n, seed=seed, push_rate=rate) policy, _ = bc.fit(s, a, seed=seed) rows.append({“labels”: len(s), **evaluate(policy)}) return {“seed”: seed, “dagger”: dagger_rows, “more”: more, “pushed”: pushed}
def part3() -> list[dict]: seeds = 1 if FAST else 3 if FAST: results = [curve(0)] else: with Pool(seeds) as pool: results = pool.map(curve, range(seeds)) for name, key in ((“DAgger (iteration: labels, success)”, “dagger”), (“more demonstrations without pushes”, “more”), (“demonstrations recorded with pushes”, “pushed”)): print(f” {name}”) for r in results: print(f” seed {r[‘seed’]}: “ + “, “.join(f”{row[‘labels’]}: {row[‘success’]:.2f}” for row in r[key])) final = {key: [r[key][-1][“success”] for r in results] for key in (“dagger”, “more”, “pushed”)} print(f” at the largest label budget, success per seed: DAgger {final[‘dagger’]}, more demonstrations {final[‘more’]}, “ f”pushed demonstrations {final[‘pushed’]}”) return results
def part4() -> None: states, actions = demonstrations(20, seed=0) policy, _ = bc.fit(states, actions, seed=0) print(f” {‘y (m)’:>8}{‘expert fy (N)’:>15}{‘clone fy (N)’:>14} at x = -0.2 m, at rest”) for y in (-0.03, -0.01, -0.003, 0.0, 0.003, 0.01, 0.03): s = np.array([-0.2, y, 0.0, 0.0]) print(f” {y:>8.3f}{pm.expert(s)[1]:>15.2f}{float(policy(s)[1]):>14.2f}”)
def figure(results: list[dict]) -> None: import matplotlib matplotlib.use(“Agg”) import matplotlib.pyplot as plt fig, ax = plt.subplots(figsize=(6.4, 3.8), constrained_layout=True) styles = ((“dagger”, “DAgger”, “#0072B2”, “o”), (“more”, “more demonstrations, no pushes”, “#D55E00”, “s”), (“pushed”, “demonstrations recorded with pushes”, “#009E73”, “^”)) for key, label, colour, marker in styles: for i, r in enumerate(results): xs = [row[“labels”] / 1000 for row in r[key]] ys = [row[“success”] for row in r[key]] ax.plot(xs, ys, color=colour, marker=marker, alpha=0.8, label=label if i == 0 else None) ax.set_xlabel(“expert labels (thousands of state-action pairs)”) ax.set_ylabel(“success with pushes (100 held-out starts)”) ax.set_ylim(0, 1.02) ax.grid(alpha=0.3) ax.legend(frameon=False, loc=”lower right”) FIGURE.parent.mkdir(parents=True, exist_ok=True) fig.savefig(FIGURE, dpi=150) plt.close(fig)
if name == “main”: print(f”torch {torch.version}; {len(HELD_OUT)} held-out starts; pushes {PUSH} per second, 0.4 m/s”) print(“(1, 2) the expert, and a clone of 20 demonstrations”) part1_2() print(“(3) success with pushes against expert labels, one line per seed”) results = part3() figure(results) print(f” figure: {FIGURE.relative_to(FIGURE.parents[2])}”) print(“(4) the expert’s choice of side, and the clone’s”) part4()
Output:
```text
torch 2.14.1+cpu; 100 held-out starts; pushes 1.0 per second, 0.4 m/s
(1, 2) the expert, and a clone of 20 demonstrations
expert without pushes success 1.00, obstacle touched 0.00, mean time 2.36 s
expert with pushes success 0.95, obstacle touched 0.11, mean time 3.19 s
20 demonstrations: 4700 state-action pairs; training MSE 0.0579 (normalized units)
clone without pushes success 1.00, obstacle touched 0.06, mean time 2.42 s
clone with pushes success 0.53, obstacle touched 0.24, mean time 3.53 s
distance from visited states to the nearest demonstrated state (units of the demonstrations' std), with pushes:
expert median 0.38, 90th percentile 1.26, 99th 2.79; largest distance per episode: successes median 2.47, failures median 3.65
clone median 1.08, 90th percentile 2.75, 99th 4.28; largest distance per episode: successes median 2.77, failures median 4.07
(3) success with pushes against expert labels, one line per seed
DAgger (iteration: labels, success)
seed 0: 4700: 0.53, 6867: 0.99, 8059: 0.94, 9385: 0.94, 10413: 0.94, 11499: 0.92
seed 1: 4718: 0.64, 6011: 0.85, 7267: 0.79, 8454: 0.93, 9774: 0.93, 10957: 0.95
seed 2: 4715: 0.59, 6241: 0.92, 7650: 0.89, 8932: 0.83, 9970: 0.85, 11409: 0.94
more demonstrations without pushes
seed 0: 4700: 0.53, 7057: 0.56, 9417: 0.54, 11769: 0.52
seed 1: 4718: 0.64, 7072: 0.52, 9423: 0.60, 11779: 0.63
seed 2: 4715: 0.59, 7066: 0.61, 9424: 0.54, 11791: 0.54
demonstrations recorded with pushes
seed 0: 7265: 0.94, 10670: 0.94, 14170: 0.95, 17365: 0.92
seed 1: 6191: 0.92, 9737: 0.93, 13147: 0.94, 16669: 0.95
seed 2: 6835: 0.95, 10673: 0.95, 14059: 0.94, 17018: 0.94
at the largest label budget, success per seed: DAgger [0.92, 0.95, 0.94], more demonstrations [0.52, 0.63, 0.54], pushed demonstrations [0.92, 0.95, 0.94]
figure: runs/l14_1/labels.png
(4) the expert's choice of side, and the clone's
y (m) expert fy (N) clone fy (N) at x = -0.2 m, at rest
-0.030 -0.68 0.04
-0.010 -0.76 0.13
-0.003 -0.79 0.17
0.000 0.80 0.18
0.003 0.79 0.19
0.010 0.76 0.22
0.030 0.68 0.29
Low validation loss, poor closed-loop success. Covariate shift: validation states come from the expert’s distribution. Measure closed-loop success, and the distance from visited states to the data.
More demonstrations do not help. They come from the same distribution. Add data where the learner goes: DAgger, or perturbed demonstrations.
The clone hesitates or drives into an obstacle where the expert would choose a side. Averaged modes. Check the expert’s actions near the decision boundary; consider a multimodal policy class.
DAgger makes things worse for an iteration. The new labels are concentrated in states the current clone visits; a policy refit on them can lose accuracy elsewhere. Keep all data (aggregate, never replace) and run more iterations.
Run DAgger with pushes at 2 per second during both rollouts and tests, 3 seeds, and report success against labels for DAgger and for clean demonstrations. How does the gap change with the rate of pushes?
Replace the MLP with a mixture density network (two Gaussian components) and repeat output (4). Show whether the clone now chooses a side near $y = 0$, and measure its obstacle-contact rate without pushes on the 100 held-out starts.
[!research] Covariate shift in learned manipulation Modern visuomotor policies trained on teleoperated demonstrations face the same problem with far larger state spaces, and the remedies in this lesson have direct descendants: on-policy corrections from a human or scripted expert (the DAgger family), noise injection during demonstration, and policy classes that represent multimodal action distributions (Chi et al., arXiv:2303.04137, in the reading list) or act in chunks to reduce compounding errors (Zhao et al., arXiv:2304.13705). An evaluation that reports only demonstration-distribution performance hides exactly the failure this lesson measures; held-out starts and perturbations at test time expose it (inference).
{"id": "14.1-check", "title": "Knowledge check", "questions": [
{"kind": "mcq", "q": "A cloned policy has low error on held-out demonstration states but fails often in closed loop. What is the most likely cause?",
"options": ["Too little training", "Covariate shift: the policy reaches states the demonstrations never covered", "The expert is wrong", "The simulation is non-deterministic"],
"answer": 1,
"explain": "<p>Output (2): visited states three times further from the data, and failures go furthest.</p>"},
{"kind": "mcq", "q": "Which change fixed the clone's failures under pushes in output (3)?",
"options": ["Recording more clean demonstrations", "Labelling the states the clone itself visits (DAgger), or recording demonstrations that include pushes", "Training longer", "A larger network"],
"answer": 1,
"explain": "<p>Both reached 0.92 to 0.95; more clean demonstrations stayed at 0.52 to 0.63.</p>"},
{"kind": "mcq", "q": "Why does the clone output a small lateral force at y = 0 where the expert outputs about ±0.8 N?",
"options": ["The network is too small", "Mean-squared-error regression of a smooth function settles between two action branches", "The data is noisy", "The expert is stochastic"],
"answer": 1,
"explain": "<p>Output (4): +0.04 to +0.29 N across ±3 cm against −0.79 and +0.80 N.</p>"},
{"kind": "mcq", "q": "What does DAgger need that cloning from recorded demonstrations does not?",
"options": ["A reward function", "An expert that can label arbitrary states the learner visits", "A simulator", "A larger dataset"],
"answer": 1,
"explain": "<p>The expert must answer queries on the learner's state distribution, which a scripted expert can and a human often cannot.</p>"}
]}
Lesson 14.2 treats demonstrations as a dataset: how to store them, how to measure what they cover, and why a validation loss is not an evaluation.