Together these are the action representation. The same motion can be written in any of them, and a learned policy only ever sees the written form. So the representation changes what function the network has to learn, which errors the controller underneath absorbs, and which changes in the world the policy survives.

This lesson measures that on the 7-DOF arm. Six representations are used to write down the same reference motions, and each is cloned with the same network, the same data size and the same training. The results are then tested under a payload, a slower control rate and targets outside the training region. One of the lesson’s findings is a correction: the first explanation for why two representations failed was wrong, and an ablation showed what was really happening. The lesson ends with action chunks on the tabletop task of Lesson 22.4.

Visual intuition

The figure shows success for each representation under the four conditions, once with the joint velocities in the policy’s observation (top) and once without them (bottom). Read the top panel first, then look at how much of it the bottom panel overturns.

Mathematics

Six ways to write one motion, and what MuJoCo runs for each

Gravity compensation always uses the nominal arm, as a real controller with a fixed model would, so a payload is unknown to every controller. The policy acts at 20 Hz. Each expert follows the same reference motion: a minimum-jerk tool path over 1.5 s, converted to joint angles by IK. Each expert’s action is whatever makes its own representation follow that motion, so the demonstrations differ only in how the motion is written. Every expert succeeds on 20 of 20 test targets. The torque expert needed gentler gains than the others: with the servo gains of the target-based controllers (400 and 40, $\omega_n$ = 20 rad/s) it succeeded on 0.92 of targets. A torque command held for 50 ms closes its feedback loop at 20 Hz, too slowly for stiff gains (see Lesson 19.1 for delay margins). With 225 and 30 it succeeded on all.

The task: bring the tool within 1.5 cm and 0.1 rad of a target pose, starting from a perturbed home pose, in 2 s. Each representation is cloned from 1000 demonstrations (40 000 frames) by an MLP with two hidden layers of 256 units, 6000 steps, three seeds, and tested on 100 held-out targets per condition:

First result: observing joint velocities

Read naively, this table says that Cartesian deltas are the right representation, that absolute targets are hopeless, and that every policy fails at 10 Hz. The two absolute representations reach the lowest training error of all (0.0001 in normalized units) and nearly never succeed. A plausible explanation offered itself: an absolute target must be predicted to a fraction of a millimetre of motion out of the whole workspace, so a small error relative to the output’s range is a large one at the tool. That explanation was wrong, as the next experiment showed.

The correction: remove the velocities

Output (1b) repeats everything with the 7 joint velocities removed from the observation (17 numbers instead of 24):

Absolute joint targets go from 0.06 to 0.89, joint velocities from 0.13 to 0.96, and the 10 Hz failures disappear. The representations were not the problem. The observation was. The arm’s joint velocities change smoothly from step to step, so the current velocity nearly determines the next action. A network can reach a low training loss by continuing the motion it is already making, instead of reading the target and computing a path to it. That shortcut works on the demonstrations and fails in closed loop, where the motion the policy is “continuing” is its own. At 10 Hz the observed velocities differ from training (the arm covers each step’s motion in twice the time), and the shortcut breaks completely.

This is a known failure of behaviour cloning. De Haan et al. call it causal misidentification and show that “access to more information can yield worse performance” (arXiv:1905.11979). Wen et al. describe the instance where expert actions are correlated over time and “the imitator learns to cheat by predicting the expert’s previous action” (arXiv:2010.14876). The torque row is the instructive exception: without velocities it collapses from 0.63 to 0.00. A torque policy has to supply the damping itself, and damping is a velocity term, so for torque the velocity is part of the cause, not a nuisance.

What remains true of the representations

With the shortcut removed, the representations still differ, and the differences now have mechanisms:

Gripper commands

The gripper of Level 10 is a position servo on a tendon (one finger driven, the other mirrored). As an action, its command is usually written either as a continuous opening or as a binary open or close. A continuous gripper output cloned from demonstrations that only ever open or close is averaged near the transitions, giving half-closed commands. Discretizing it, as RT-1 does for every action dimension (256 bins, arXiv:2212.06817), or thresholding a continuous output, makes the command decisive (inference; Lesson 10.1 measures what the squeeze force of a closed command must carry). The pushing task of this level has no gripper.

Action chunks

A policy can output a sequence of the next $H$ actions instead of one. ACT (arXiv:2304.13705), Octo (arXiv:2405.12213) and π0 (arXiv:2410.24164) all do; π0 uses chunks of 50. A chunk can be executed three ways: apply only its first action and predict again (receding horizon); apply the whole chunk open loop; or average, for each step, every chunk that predicted it (ACT’s temporal ensemble). Output (2) measures this on the tabletop pushing task with state policies, 2000 demonstrations, three seeds and 100 held-out tasks:

Here chunks cost success, and executing them open loop costs most. Pushing a round puck needs a correction at every step: the demonstrator corrects the pusher’s sideways offset every 0.1 s, and a chunk executed open loop gives that up for 0.4 or 0.8 s. The demonstrator is a deterministic function of the current state, so there is no multimodality or hidden state for a chunk to resolve. That is the situation in which chunking helps in the papers above, which learn from human demonstrations (inference). Whether chunking helps is a property of the task and the data, to be measured, not assumed.

Implementation

Representation	One action means	Between policy and motors, at 500 Hz	In MuJoCo
torque	joint torques, fractions of each motor’s limit	nothing	`<motor>` actuators, `ctrl` = torque
absolute joint position	joint targets, fractions of each joint’s range	PD + gravity compensation	`<position>` actuators, or PD on `qfrc_bias`
joint delta	target increments of up to 0.05 rad	the same servo, on an integrated target	as above
joint velocity	joint velocities of up to 1 rad/s	a velocity servo + gravity compensation	`<velocity>` actuators
Cartesian delta	tool-target increments: 2 cm in x, y, z and 0.1 rad of yaw	differential IK (Lesson 6.2), then PD	IK on `mj_jacSite`, then a servo
absolute Cartesian pose	the tool target in the workspace box, and yaw	differential IK, then PD	as above

Representation	nominal	payload 1 kg	10 Hz	new region	final error, nominal (mm)
torque	0.63	0.00	0.00	0.11	12.8
absolute joint position	0.06	0.00	0.03	0.00	42.6
joint delta	0.47	0.41	0.01	0.08	15.9
joint velocity	0.13	0.00	0.09	0.01	29.8
Cartesian delta	0.97	0.83	0.03	0.20	5.1
absolute Cartesian pose	0.01	0.00	0.00	0.00	49.4

Representation	nominal	payload 1 kg	10 Hz	new region	final error, nominal (mm)
torque	0.00	0.00	0.00	0.00	253.3
absolute joint position	0.89	0.02	0.92	0.01	7.1
joint delta	0.74	0.70	0.65	0.22	9.4
joint velocity	0.96	0.70	0.97	0.49	4.1
Cartesian delta	0.99	0.84	1.00	0.59	3.8
absolute Cartesian pose	0.85	0.01	0.91	0.06	7.9

Chunk length	first action only	whole chunk, open loop	temporal ensemble
1	0.90
4	0.87	0.75	0.79
8	0.84	0.47	0.49

INPUT: arm7.xml through mjcourse.vla.actions; the tabletop scene for chunks
PROCESS: (1) six representations, BC, 3 seeds, 4 conditions; (1b) the same without joint velocities; (2) chunks on the tabletop
OUTPUT: tables; runs/l22_3/representations.png; runs/l22_3/results.json

```python file=examples/l22_3_action_representation.py import json import multiprocessing import os from concurrent.futures import ProcessPoolExecutor from pathlib import Path

FAST = os.environ.get(“MJC_FAST”) == “1” SEEDS = (0,) if FAST else (0, 1, 2) DEMOS, TESTS = (40, 10) if FAST else (1000, 100) TRAIN_LOW, TRAIN_HIGH = np.array([0.35, -0.25, 0.15]), np.array([0.60, 0.10, 0.45]) NEW_LOW, NEW_HIGH = np.array([0.35, 0.15, 0.15]), np.array([0.60, 0.25, 0.45]) CONDITIONS = (“nominal”, “payload 1 kg”, “10 Hz”, “new region”) OUT = Path(file).resolve().parents[1] / “runs” / “l22_3”

env, rng = A.ArmReach(rep), np.random.default_rng(0)
expert = A.Expert(env)
obs, act = [], []
for _ in range(DEMOS):
    env.reset(A.start_pose(rng, env.home), A.sample_target(rng, TRAIN_LOW, TRAIN_HIGH))
    expert.plan()
    for _ in range(A.STEPS):
        obs.append(env.obs())
        act.append(expert.act())
        env.step(act[-1])
return np.array(obs), np.array(act)

def evaluate(policy, rep: str, condition: str) -> dict: from mjcourse.vla import actions as A

env = A.ArmReach(rep, payload=1.0 if condition == "payload 1 kg" else 0.0,
                 frame_skip=2 * A.FRAME_SKIP if condition == "10 Hz" else A.FRAME_SKIP)
steps = A.STEPS                                                          # 40 decisions in every condition
low, high = (NEW_LOW, NEW_HIGH) if condition == "new region" else (TRAIN_LOW, TRAIN_HIGH)
rng = np.random.default_rng(1000)                                        # held-out starts and targets
ok, pos_err, jerk = [], [], []
for _ in range(TESTS):
    env.reset(A.start_pose(rng, env.home), A.sample_target(rng, low, high))
    prev = None
    for _ in range(steps):
        a = np.clip(policy(env.obs()), -1, 1)
        if prev is not None:
            jerk.append(np.abs(a - prev).mean())
        prev = a
        env.step(a)
    ok.append(env.success())
    pos_err.append(env.errors()[0])
return {"success": float(np.mean(ok)), "pos_err": float(np.median(pos_err)), "jerk": float(np.mean(jerk))}

torch.set_num_threads(1)
from mjcourse.il import bc

obs, act = demonstrations(rep)
keep = KEEP_NO_VELOCITY if drop_velocity else np.ones(obs.shape[1], bool)
policy, info = bc.fit(obs[:, keep], act, steps=600 if FAST else 6000, hidden=256, seed=seed)
return rep, seed, info["train_mse"], {c: evaluate(lambda o: policy(o[keep]), rep, c) for c in CONDITIONS}

def part1(pool, drop_velocity: bool = False) -> dict: from mjcourse.vla.actions import REPRESENTATIONS

label = "(1b) without joint velocities in the observation" if drop_velocity else "(1) six action representations"
print(f"{label}, the same reference motions; behaviour cloning, "
      f"{DEMOS} demonstrations, {len(SEEDS)} seed(s), {TESTS} held-out targets per condition")
results = list(pool.map(part1_job, [(r, s, drop_velocity) for r in REPRESENTATIONS for s in SEEDS]))
print(f"  {'representation':<17}" + "".join(f"{c:>24}" for c in CONDITIONS) + f"{'train MSE':>11}")
for rep in REPRESENTATIONS:
    rows = [r for r in results if r[0] == rep]
    cells = []
    for c in CONDITIONS:
        vals = [r[3][c]["success"] for r in rows]
        cells.append(f"{np.mean(vals):.2f} ({', '.join(f'{v:.2f}' for v in vals)})")
    print(f"  {rep:<17}" + "".join(f"{x:>24}" for x in cells) + f"{np.mean([r[2] for r in rows]):>11.4f}")
print("  median final position error (mm), nominal and payload; mean |change of action| per step (smoothness), nominal")
for rep in REPRESENTATIONS:
    rows = [r for r in results if r[0] == rep]
    print(f"  {rep:<17} {1000 * np.mean([r[3]['nominal']['pos_err'] for r in rows]):7.1f}"
          f" {1000 * np.mean([r[3]['payload 1 kg']['pos_err'] for r in rows]):7.1f}"
          f"   {np.mean([r[3]['nominal']['jerk'] for r in rows]):.3f}")
return {rep: {c: [r[3][c]["success"] for r in results if r[0] == rep] for c in CONDITIONS} for rep in REPRESENTATIONS}

def figure(tables: dict) -> None: import matplotlib matplotlib.use(“Agg”) import matplotlib.pyplot as plt

from mjcourse.vla.actions import REPRESENTATIONS

OUT.mkdir(parents=True, exist_ok=True)
(OUT / "results.json").write_text(json.dumps(tables, indent=1))
fig, axes = plt.subplots(len(tables), 1, figsize=(9, 3.3 * len(tables)), constrained_layout=True, sharex=True)
axes = np.atleast_1d(axes)
colours = ("#0072B2", "#D55E00", "#009E73", "#CC79A7")
width = 0.2
for ax, (title, table) in zip(axes, tables.items()):
    for j, (c, colour) in enumerate(zip(CONDITIONS, colours)):
        means = [np.mean(table[r][c]) for r in REPRESENTATIONS]
        xs = np.arange(len(REPRESENTATIONS)) + (j - 1.5) * width
        ax.bar(xs, means, width, color=colour, label=c)
        for x, r in zip(xs, REPRESENTATIONS):
            ax.plot([x] * len(table[r][c]), table[r][c], "k.", markersize=3)
    ax.set_ylabel("success (100 targets)")
    ax.set_ylim(0, 1.05)
    ax.set_title(f"observation {title}", fontsize=10)
    ax.grid(axis="y", alpha=0.3)
handles, labels = axes[0].get_legend_handles_labels()
fig.legend(handles, labels, frameon=False, ncol=4, loc="outside upper center", fontsize=8)
axes[-1].set_xticks(np.arange(len(REPRESENTATIONS)), [r.replace("_", " ") for r in REPRESENTATIONS])
fig.savefig(OUT / "representations.png", dpi=150)
plt.close(fig)

from mjcourse import tabletop as tt

torch.set_num_threads(1)
n_demos = 300 if FAST else 2000

def onehot(task):
    v = np.zeros(7, np.float32)
    v[tt.OBJECTS.index(task[0])] = 1
    v[3 + list(tt.ZONES).index(task[1])] = 1
    return v

xs, ys = [], []
for i in range(n_demos):
    rng = np.random.default_rng([0, i])
    task = (tt.OBJECTS[rng.integers(3)], list(tt.ZONES)[rng.integers(4)])
    scene = tt.Tabletop()
    scene.reset(tt.sample_layout(rng, task))
    states, acts = [], []
    for _ in range(tt.EPISODE_STEPS):
        s = scene.state()
        states.append(np.r_[s, onehot(task)])
        acts.append(tt.expert(s, task))
        scene.step(acts[-1])
        if scene.success(task):
            break
    if not scene.success(task):
        continue
    padded = np.concatenate([acts, np.zeros((chunk, 2))])
    for t in range(len(states)):
        xs.append(states[t])
        ys.append(padded[t:t + chunk].ravel())
torch.manual_seed(seed)
x, y = torch.as_tensor(np.array(xs, np.float32)), torch.as_tensor(np.array(ys, np.float32))
net = nn.Sequential(nn.Linear(23, 256), nn.ReLU(), nn.Linear(256, 256), nn.ReLU(), nn.Linear(256, 2 * chunk), nn.Tanh())
opt = torch.optim.Adam(net.parameters(), 1e-3)
for _ in range(1500 if FAST else 15000):
    idx = torch.randint(0, len(x), (256,))
    loss = ((net(x[idx]) - y[idx]) ** 2).mean()
    opt.zero_grad()
    loss.backward()
    opt.step()
out = {}
for mode in ("first action", "open-loop chunk", "temporal ensemble"):
    if chunk == 1 and mode != "first action":
        continue
    ok = []
    for i in range(30 if FAST else 100):
        rng = np.random.default_rng([99, i])
        task = (tt.OBJECTS[rng.integers(3)], list(tt.ZONES)[rng.integers(4)])
        scene = tt.Tabletop()
        scene.reset(tt.sample_layout(rng, task))
        history, plan = [], []
        for t in range(tt.EPISODE_STEPS):
            with torch.no_grad():
                c = net(torch.as_tensor(np.r_[scene.state(), onehot(task)], dtype=torch.float32)).numpy().reshape(chunk, 2)
            if mode == "first action":
                a = c[0]
            elif mode == "open-loop chunk":
                if not plan:
                    plan = list(c)
                a = plan.pop(0)
            else:
                history = [(t0, h) for t0, h in history if t - t0 < chunk] + [(t, c)]
                a = np.mean([h[t - t0] for t0, h in history], axis=0)
            scene.step(a)
            if scene.success(task):
                break
        ok.append(scene.success(task))
    out[mode] = float(np.mean(ok))
return chunk, seed, out

def part2(pool) -> None: print(f”(2) action chunks on the tabletop: state policies, {300 if FAST else 2000} demonstrations, “ f”success on {30 if FAST else 100} held-out tasks”) results = list(pool.map(part2_job, [(c, s) for c in (1, 4, 8) for s in SEEDS])) for chunk in (1, 4, 8): for mode in (“first action”, “open-loop chunk”, “temporal ensemble”): vals = [r[2][mode] for r in results if r[0] == chunk and mode in r[2]] if vals: print(f” chunk {chunk}, {mode:<18} {np.mean(vals):.2f} ({‘, ‘.join(f’{v:.2f}’ for v in vals)})”)

if name == “main”: with ProcessPoolExecutor(min(32, os.cpu_count() or 1), mp_context=multiprocessing.get_context(“spawn”)) as pool: part, tables = os.environ.get(“MJC_PART”, “all”), {} if part in (“all”, “1”): tables[“with joint velocities”] = part1(pool) if part in (“all”, “1b”): tables[“without joint velocities”] = part1(pool, drop_velocity=True) if tables: figure(tables) if part in (“all”, “2”): part2(pool)

The task, the six controllers and the experts:

```python file=src/mjcourse/vla/actions.py
from __future__ import annotations

from dataclasses import dataclass

import mujoco
import numpy as np

from mjcourse import kinematics, spatial
from mjcourse.paths import model_path

REPRESENTATIONS = ("torque", "joint_position", "joint_delta", "joint_velocity", "cartesian_delta", "cartesian_pose")
FRAME_SKIP, STEPS, MOVE_STEPS = 25, 40, 30            # 20 Hz policy, 2 s episodes, 1.5 s reference motion
BOX_LOW, BOX_HIGH = np.array([0.35, -0.25, 0.15]), np.array([0.60, 0.25, 0.45])
YAW_RANGE = 0.6
JOINT_STEP, JOINT_SPEED, EE_STEP, YAW_STEP = 0.05, 1.0, 0.02, 0.1
SUCCESS_POS, SUCCESS_YAW = 0.015, 0.1
OBS_DIM = 24


@dataclass(frozen=True)
class Target:
    pos: np.ndarray
    yaw: float


def sample_target(rng: np.random.Generator, low=BOX_LOW, high=BOX_HIGH) -> Target:
    return Target(rng.uniform(low, high), float(rng.uniform(-YAW_RANGE, YAW_RANGE)))


def _yaw_of(mat: np.ndarray) -> float:
    return float(np.arctan2(mat[1, 0], mat[0, 0]))


class ArmReach:
    def __init__(self, representation: str, payload: float = 0.0, frame_skip: int = FRAME_SKIP):
        if representation not in REPRESENTATIONS:
            raise ValueError(representation)
        self.rep, self.frame_skip = representation, frame_skip
        spec = mujoco.MjSpec.from_file(str(model_path("arm7")))
        if payload:
            body = spec.body("link7")
            body.add_geom(type=mujoco.mjtGeom.mjGEOM_SPHERE, size=[0.03, 0, 0], pos=[0, 0, 0.1], mass=payload,
                          contype=0, conaffinity=0, rgba=[0.9, 0.6, 0.1, 1])
        self.model = spec.compile()
        self.data = mujoco.MjData(self.model)
        # Gravity compensation uses the nominal arm, as a real controller with a fixed model would: a payload
        # is something the controller does not know about.
        self.nominal = mujoco.MjModel.from_xml_path(str(model_path("arm7")))
        self.nominal_data = mujoco.MjData(self.nominal)
        m = self.model
        self.site = m.site("ee").id
        self.lo, self.hi = m.jnt_range[:7].T
        self.limit = m.actuator_ctrlrange[:7, 1]
        mujoco.mj_resetDataKeyframe(m, self.data, m.key("home").id)
        mujoco.mj_forward(m, self.data)
        self.home = self.data.qpos[:7].copy()
        self.down = spatial.mat_to_quat(self.data.site_xmat[self.site].reshape(3, 3))     # tool down, yaw of home
        self.home_yaw = _yaw_of(self.data.site_xmat[self.site].reshape(3, 3))
        mass = np.zeros((m.nv, m.nv))
        mujoco.mj_fullM(m, self.data, mass)
        inertia = np.diag(mass)[:7]
        self.kp, self.kd = inertia * 400.0, inertia * 40.0

    # ------------------------------------------------------------------ state and observation
    def reset(self, q0: np.ndarray, target: Target) -> None:
        mujoco.mj_resetData(self.model, self.data)
        self.data.qpos[:7] = q0
        mujoco.mj_forward(self.model, self.data)
        self.target = target
        self.q_target = q0.copy()
        self.x_target = self.data.site_xpos[self.site].copy()
        self.yaw_target = self.yaw()

    def yaw(self) -> float:
        raw = _yaw_of(self.data.site_xmat[self.site].reshape(3, 3)) - self.home_yaw
        return float((raw + np.pi) % (2 * np.pi) - np.pi)

    def obs(self) -> np.ndarray:
        d, y, ty = self.data, self.yaw(), self.target.yaw
        return np.concatenate([d.qpos[:7], d.qvel[:7], d.site_xpos[self.site], [np.sin(y), np.cos(y)],
                               self.target.pos, [np.sin(ty), np.cos(ty)]]).astype(np.float32)

    def errors(self) -> tuple[float, float]:
        pos = float(np.linalg.norm(self.data.site_xpos[self.site] - self.target.pos))
        yaw = abs((self.yaw() - self.target.yaw + np.pi) % (2 * np.pi) - np.pi)
        return pos, yaw

    def success(self) -> bool:
        pos, yaw = self.errors()
        return pos < SUCCESS_POS and yaw < SUCCESS_YAW

    # ------------------------------------------------------------------ executing one action
    def _tool_quat(self, yaw: float) -> np.ndarray:
        return spatial.quat_mul(np.array([np.cos(yaw / 2), 0, 0, np.sin(yaw / 2)]), self.down)

    def _ik(self, pos: np.ndarray, yaw: float) -> np.ndarray:
        result = kinematics.solve_ik(self.model, "ee", pos, self._tool_quat(yaw), q_init=np.r_[self.q_target],
                                     joints=list(range(7)), max_iters=20)
        return np.clip(result.qpos[:7], self.lo, self.hi)

    def step(self, action: np.ndarray) -> None:
        a = np.clip(np.asarray(action, dtype=float), -1.0, 1.0)
        rep = self.rep
        if rep == "joint_position":
            self.q_target = self.lo + (a + 1) / 2 * (self.hi - self.lo)
        elif rep == "joint_delta":
            self.q_target = np.clip(self.q_target + JOINT_STEP * a, self.lo, self.hi)
        elif rep == "cartesian_delta":
            self.x_target = np.clip(self.x_target + EE_STEP * a[:3], BOX_LOW - 0.05, BOX_HIGH + 0.05)
            self.yaw_target = self.yaw_target + YAW_STEP * a[3]
            self.q_target = self._ik(self.x_target, self.yaw_target)
        elif rep == "cartesian_pose":
            self.x_target = BOX_LOW - 0.05 + (a[:3] + 1) / 2 * (BOX_HIGH - BOX_LOW + 0.1)
            self.yaw_target = a[3] * YAW_RANGE * 1.2
            self.q_target = self._ik(self.x_target, self.yaw_target)
        m, d, nd = self.model, self.data, self.nominal_data
        for _ in range(self.frame_skip):
            mujoco.mj_forward(m, d)
            nd.qpos[:], nd.qvel[:] = d.qpos[:7], d.qvel[:7]
            mujoco.mj_forward(self.nominal, nd)
            bias = nd.qfrc_bias[:7]                                        # the nominal model's gravity and Coriolis
            if rep == "torque":
                tau = a * self.limit
            elif rep == "joint_velocity":
                tau = bias + self.kd * 2 * (JOINT_SPEED * a - d.qvel[:7])
            else:
                tau = bias + self.kp * (self.q_target - d.qpos[:7]) - self.kd * d.qvel[:7]
            d.ctrl[:7] = np.clip(tau, -self.limit, self.limit)
            mujoco.mj_step(m, d)

    @property
    def action_dim(self) -> int:
        return 4 if self.rep.startswith("cartesian") else 7


def min_jerk(s: float) -> float:
    s = min(max(s, 0.0), 1.0)
    return 10 * s**3 - 15 * s**4 + 6 * s**5


class Expert:
    def __init__(self, env: ArmReach):
        self.env = env

    def plan(self) -> None:
        env = self.env
        x0, y0 = env.data.site_xpos[env.site].copy(), env.yaw()
        q = env.data.qpos[:7].copy()
        self.x_ref, self.yaw_ref, self.q_ref = [], [], []
        saved = env.q_target.copy()
        for k in range(STEPS + 2):
            s = min_jerk(k / MOVE_STEPS)
            x = x0 + s * (env.target.pos - x0)
            yaw = y0 + s * ((env.target.yaw - y0 + np.pi) % (2 * np.pi) - np.pi)
            env.q_target = q
            q = env._ik(x, yaw)
            self.x_ref.append(x)
            self.yaw_ref.append(yaw)
            self.q_ref.append(q)
        env.q_target = saved
        self.k = 0

    def act(self) -> np.ndarray:
        env, k = self.env, self.k
        self.k += 1
        q_next, x_next, y_next = self.q_ref[k + 1], self.x_ref[k + 1], self.yaw_ref[k + 1]
        dt = env.frame_skip * env.model.opt.timestep
        rep = env.rep
        if rep == "joint_position":
            a = 2 * (q_next - env.lo) / (env.hi - env.lo) - 1
        elif rep == "joint_delta":
            a = (q_next - env.q_target) / JOINT_STEP
        elif rep == "joint_velocity":
            a = ((q_next - env.data.qpos[:7]) / dt) / JOINT_SPEED
        elif rep == "cartesian_delta":
            a = np.r_[(x_next - env.x_target) / EE_STEP, (y_next - env.yaw_target) / YAW_STEP]
        elif rep == "cartesian_pose":
            a = np.r_[2 * (x_next - (BOX_LOW - 0.05)) / (BOX_HIGH - BOX_LOW + 0.1) - 1, y_next / (YAW_RANGE * 1.2)]
        else:                                                               # torque: computed torque on the reference
            m, d = env.model, env.data
            mujoco.mj_forward(m, d)
            mass = np.zeros((m.nv, m.nv))
            mujoco.mj_fullM(m, d, mass)
            qd_ref = (q_next - self.q_ref[k]) / dt
            acc = 225.0 * (q_next - d.qpos[:7]) + 30.0 * (qd_ref - d.qvel[:7])    # 15 rad/s: a 20 Hz torque loop cannot be stiffer
            a = (mass[:7, :7] @ acc + d.qfrc_bias[:7]) / env.limit
        return np.clip(a, -1.0, 1.0)


def start_pose(rng: np.random.Generator, home: np.ndarray) -> np.ndarray:
    return home + rng.uniform(-0.15, 0.15, 7)

Debugging

A cloned policy has a tiny training loss and fails in closed loop. Look for a shortcut: an input that predicts the expert’s action on the demonstrations without being its cause (here the joint velocities). Ablate it and compare closed-loop success, not loss.

A policy works at one control rate and not at another. Check both the action’s meaning (increments scale with the step, velocities do not) and the observation’s distribution (velocities and accelerations change with the rate).

A torque policy sags under load. It has learned the nominal arm’s gravity torques. Any representation with a servo underneath absorbs part of a load error; a torque policy absorbs none.

Exercise

Repeat output (1b) with 300 demonstrations instead of 1000. Predict, before running it, which representation loses most.

Challenge

Give the torque representation a velocity input again, but as the difference of the last two joint positions divided by the control step, rather than the simulator’s instantaneous velocity. Does the shortcut return? Then add Gaussian noise of increasing size to that input, and find the noise level at which the torque policy’s damping and the shortcut trade off.

Research connection

Lesson 22.4 builds the MiniVLA, a policy from pixels and words to pusher velocities, and meets the same shortcut again.

Core concept