Core concept

PPO learns only from data its current policy collected, and throws that data away after a few epochs. Off-policy actor-critic methods keep every transition in a replay buffer and learn from it repeatedly: a critic $Q(s, a)$ is fit to bootstrapped targets, an actor is trained to choose actions the critic rates highly, and slowly updated copies of both (target networks) keep the targets from chasing themselves. DDPG is the basic form; TD3 and SAC add specific repairs. The price is computation per environment step and more ways to fail; the gain is that far fewer environment steps are needed.

Comparing any of these methods, or any two configurations of one, is an experiment, and the second half of this lesson is how to run it: evaluate the deterministic policy on initial states never used in training, treat seeds as the unit of replication, report an interval, and choose metrics that still discriminate when success saturates. The Level 13 checkpoint is that experiment for PPO with and without observation normalization.

[!established] What TD3 and SAC add to DDPG DDPG (Lillicrap et al., arXiv:1509.02971) trains a deterministic actor through a single critic, exploring with added noise. TD3 (Fujimoto et al., arXiv:1802.09477) learns two critics and uses the smaller of their target values (clipped double Q-learning) against overestimation, adds clipped noise to the target action (target policy smoothing), and updates the actor less often than the critics. SAC (Haarnoja et al., arXiv:1801.01290 and arXiv:1812.05905) uses a stochastic actor and adds the policy’s entropy to the objective with a weight $\alpha$, tuned automatically toward a target entropy, together with two critics. mjcourse.rl.offpolicy implements all three in one learner with these as switches.

Visual intuition

This lesson has no lab; its picture is the first table of output (1, 2). Three seeds of SAC reach every held-out target within 30 000 environment steps; three seeds of PPO need about 55 000; and the same SAC with one number changed (the initial entropy weight) has learned almost nothing by 40 000.

Mathematics

The critic targets

For a transition $(s, a, r, s’, d)$ with $d = 1$ only if the episode terminated (never for truncation):

where primes mark target networks. SAC’s actor minimizes $\mathbb{E}[\alpha\log\pi(a\mid s) - \min_i Q_i(s, a)]$ through a reparameterized tanh-Gaussian sample, and $\alpha$ follows gradient steps that push the policy’s entropy toward $-\dim(a)$.

Off-policy against PPO on the reach task

Output (1, 2): 40 000 environment steps for the off-policy methods (one environment, one gradient step per environment step, the first 2 000 steps random), deterministic success on 20 held-out initial states every 10 000 steps, then on 100:

PPO with 16 environments reaches 0.85 to 1.00 at 51 000 steps and 1.00 by 71 000 in all three seeds, in 19 s for 100 000 steps.

Three readings. SAC is the most sample-efficient here, by a factor of about two over PPO; PPO is the fastest by the clock, by a factor of more than ten, because it steps 16 environments in parallel and makes one batch of updates per 2 048 transitions instead of one per transition. Which one wins depends on whether samples or seconds are expensive: on a real robot, samples; in simulation, usually seconds.

Second, SAC’s initial $\alpha$ matters as much as the algorithm. The entropy term is $\alpha$ times an entropy whose scale grows with the action dimension (7 here), while the reward per step is about −0.3; with $\alpha = 1$ the policy is rewarded mostly for being random, and the automatic tuning needs far more than 40 000 steps to bring $\alpha$ down (inference: $\alpha$ and the entropy were not logged). The temperature has to be read against the reward’s scale; a fixed default is a hidden assumption about it.

Third, TD3 learned more slowly than DDPG on this task, in every seed. Its repairs address overestimation and instability of the critic; on a short, dense, easy task they cost speed (the actor updates half as often and the clipped double Q is pessimistic) without a failure to repair (inference: the critics’ values were not logged to show it). An improvement measured on one benchmark is a statement about that benchmark.

The checkpoint: observation normalization, measured

Output (3): PPO with and without running observation normalization, 5 seeds each, 300 000 steps, evaluated deterministically on 100 held-out initial states. Success is 1.00 for every seed of both: the task saturates it, so success cannot distinguish them, and a comparison reported on success alone would conclude “no difference”. Two other measures still discriminate:

(Means over seeds with 95% percentile bootstrap intervals; the steps-until-90% values are quantized to the evaluation points, every 10 240 steps.) With normalization the policy reaches slightly faster and learning needs 21% fewer environment steps (53.2k against 67.6k), and both intervals of the paired differences exclude zero. Interquartile means agree with the means (6.94 and 7.36; 54.6k and 68.3k).

Evaluation, done so the numbers mean something

• Deterministic policy, held-out states. Training success comes from the stochastic policy on the states it trained on. Here, after 300 000 steps, both are 1.00 for every seed, but during learning they differ, and only the held-out deterministic number says what a deployed policy does.
• Seeds are the unit. Episodes within one seed share one trained network, so 500 episodes from 5 seeds are not 500 independent samples of “how good is the method”. Compute the statistic per seed, then the interval over seeds.
• Few seeds give misleading intervals. Output (4) bootstraps the mean steps-to-reach from the first 2, 3, 4 and 5 seeds: the interval from 2 seeds (width 0.08) is narrower than from 5 (0.14), because two values give the resampler almost nothing to vary. A percentile bootstrap over 2 or 3 seeds reports overconfidence, not certainty.
• Choose metrics that do not saturate. Report time-to-threshold, path length, final error or a harder held-out set alongside success.

[!recommendation] A minimum reporting standard for RL results in this course Five or more seeds per configuration; deterministic evaluation on at least 100 held-out initial states fixed in advance; per-seed values printed, then a mean or IQM with a bootstrap interval over seeds; paired differences when configurations share seeds; at least one metric that does not saturate; and the configuration of every run.

Implementation

INPUT: ReachEnv with joint-target actions; mjcourse.rl.offpolicy and mjcourse.rl.ppo
PROCESS: DDPG, TD3 and two SAC temperatures for 40 000 steps; PPO with periodic held-out evaluation; PPO with and without observation normalization, 5 seeds; interval width against the number of seeds
OUTPUT: printed tables

```python file=src/mjcourse/rl/offpolicy.py “"”DDPG, TD3 and SAC in one learner (Lesson 13.3), so that what each adds is a switch.

All three keep a replay buffer, learn a critic Q(s, a) by bootstrapped targets and an actor that climbs it, and move slow-moving target networks toward the learned ones. DDPG deterministic actor, Gaussian exploration noise, one critic. TD3 adds two critics with the minimum in the target (clipped double Q), noise on the target action (target policy smoothing) and an actor updated every second step. SAC a stochastic tanh-Gaussian actor, two critics with the minimum, and an entropy term whose weight alpha is tuned to keep the policy’s entropy near a target. Truncation is not a terminal state: the target bootstraps unless terminated is true. “””

from future import annotations

import copy import time from dataclasses import dataclass

import numpy as np import torch from torch import nn from torch.nn import functional as F

@dataclass class OffPolicyConfig: algo: str = “sac” # “ddpg”, “td3” or “sac” total_steps: int = 60_000 start_steps: int = 2_000 # uniform random actions before learning starts batch: int = 256 buffer: int = 200_000 gamma: float = 0.99 tau: float = 0.005 # target network update rate lr: float = 3e-4 hidden: tuple[int, …] = (256, 256) explore_noise: float = 0.1 # DDPG and TD3 target_noise: float = 0.2 # TD3 smoothing noise, clipped at 0.5 policy_delay: int = 2 # TD3 init_alpha: float = 0.1 # SAC’s initial entropy weight; compare with the reward scale seed: int = 0 eval_every: int = 10_000

def net(sizes) -> nn.Sequential: layers = [] for a, b in zip(sizes[:-1], sizes[1:]): layers += [nn.Linear(a, b), nn.ReLU()] return nn.Sequential(*layers[:-1])

class Actor(nn.Module): def init(self, obs_dim, act_dim, hidden, stochastic: bool): super().init() self.body = net((obs_dim, *hidden, act_dim * (2 if stochastic else 1))) self.stochastic, self.act_dim = stochastic, act_dim

def forward(self, obs, deterministic: bool = False):
    """Action in [-1, 1] and, for the stochastic actor, its log-probability."""
    out = self.body(obs)
    if not self.stochastic:
        return torch.tanh(out), None
    mean, log_std = out[..., :self.act_dim], out[..., self.act_dim:].clamp(-5, 2)
    if deterministic:
        return torch.tanh(mean), None
    normal = torch.distributions.Normal(mean, log_std.exp())
    raw = normal.rsample()
    action = torch.tanh(raw)
    log_prob = (normal.log_prob(raw) - torch.log(1 - action.pow(2) + 1e-6)).sum(-1)   # change of variables
    return action, log_prob

class Critic(nn.Module): def init(self, obs_dim, act_dim, hidden, twin: bool): super().init() self.q = nn.ModuleList([net((obs_dim + act_dim, *hidden, 1)) for _ in range(2 if twin else 1)])

def forward(self, obs, act) -> list[torch.Tensor]:
    x = torch.cat([obs, act], -1)
    return [q(x).squeeze(-1) for q in self.q]

class Learner: def init(self, obs_dim: int, act_dim: int, cfg: OffPolicyConfig): torch.manual_seed(cfg.seed) self.cfg, self.act_dim = cfg, act_dim self.actor = Actor(obs_dim, act_dim, cfg.hidden, stochastic=cfg.algo == “sac”) self.critic = Critic(obs_dim, act_dim, cfg.hidden, twin=cfg.algo != “ddpg”) self.actor_target, self.critic_target = copy.deepcopy(self.actor), copy.deepcopy(self.critic) self.actor_opt = torch.optim.Adam(self.actor.parameters(), lr=cfg.lr) self.critic_opt = torch.optim.Adam(self.critic.parameters(), lr=cfg.lr) self.log_alpha = torch.full((1,), float(np.log(cfg.init_alpha)), requires_grad=True) self.alpha_opt = torch.optim.Adam([self.log_alpha], lr=cfg.lr) self.target_entropy = -float(act_dim) self.updates = 0

@torch.no_grad()
def act(self, obs: np.ndarray, explore: bool) -> np.ndarray:
    a, _ = self.actor(torch.as_tensor(obs, dtype=torch.float32), deterministic=not explore)
    a = a.numpy()
    if explore and self.cfg.algo != "sac":
        a = a + self.cfg.explore_noise * np.random.standard_normal(a.shape)
    return np.clip(a, -1, 1)

def update(self, o, a, r, o2, term) -> None:
    cfg = self.cfg
    with torch.no_grad():
        if cfg.algo == "sac":
            a2, logp2 = self.actor(o2)
            q2 = torch.min(*self.critic_target(o2, a2)) - self.log_alpha.exp() * logp2
        else:
            a2, _ = self.actor_target(o2)
            if cfg.algo == "td3":
                a2 = (a2 + (cfg.target_noise * torch.randn_like(a2)).clamp(-0.5, 0.5)).clamp(-1, 1)
            qs = self.critic_target(o2, a2)
            q2 = torch.min(*qs) if len(qs) == 2 else qs[0]
        target = r + cfg.gamma * (1 - term) * q2
    critic_loss = sum(F.mse_loss(q, target) for q in self.critic(o, a))
    self.critic_opt.zero_grad()
    critic_loss.backward()
    self.critic_opt.step()
    self.updates += 1
    if cfg.algo == "td3" and self.updates % cfg.policy_delay:
        return
    if cfg.algo == "sac":
        a_new, logp = self.actor(o)
        actor_loss = (self.log_alpha.exp().detach() * logp - torch.min(*self.critic(o, a_new))).mean()
    else:
        actor_loss = -self.critic(o, self.actor(o)[0])[0].mean()
    self.actor_opt.zero_grad()
    actor_loss.backward()
    self.actor_opt.step()
    if cfg.algo == "sac":
        alpha_loss = -(self.log_alpha * (logp.detach() + self.target_entropy)).mean()
        self.alpha_opt.zero_grad()
        alpha_loss.backward()
        self.alpha_opt.step()
    with torch.no_grad():
        for net_, target_ in ((self.actor, self.actor_target), (self.critic, self.critic_target)):
            for p, tp in zip(net_.parameters(), target_.parameters()):
                tp.mul_(1 - cfg.tau).add_(cfg.tau * p)

def train(make_env, cfg: OffPolicyConfig, evaluate_fn=None, log=print) -> tuple[Learner, list[dict]]: “"”One environment, one gradient update per environment step after start_steps.””” env = make_env() env.action_space.seed(cfg.seed) # the warm-up actions come from the space’s own generator np.random.seed(cfg.seed) obs_dim, act_dim = env.observation_space.shape[0], env.action_space.shape[0] learner = Learner(obs_dim, act_dim, cfg) buf = {k: np.zeros((cfg.buffer, n), np.float32) for k, n in ((“o”, obs_dim), (“a”, act_dim), (“o2”, obs_dim))} buf |= {“r”: np.zeros(cfg.buffer, np.float32), “term”: np.zeros(cfg.buffer, np.float32)} size, history, start = 0, [], time.perf_counter() obs, _ = env.reset(seed=cfg.seed) for step in range(1, cfg.total_steps + 1): action = env.action_space.sample() if step <= cfg.start_steps else learner.act(obs, explore=True) obs2, reward, terminated, truncated, _ = env.step(action.astype(np.float32)) i = size % cfg.buffer buf[“o”][i], buf[“a”][i], buf[“r”][i], buf[“o2”][i], buf[“term”][i] = obs, action, reward, obs2, float(terminated) size += 1 obs = obs2 if terminated or truncated: obs, _ = env.reset() if step > cfg.start_steps: idx = np.random.randint(0, min(size, cfg.buffer), cfg.batch) learner.update(*(torch.as_tensor(buf[k][idx]) for k in (“o”, “a”, “r”, “o2”, “term”))) if evaluate_fn is not None and step % cfg.eval_every == 0: row = {“steps”: step, “seconds”: time.perf_counter() - start, **evaluate_fn(learner)} history.append(row) if log: log(row) env.close() return learner, history

```python file=examples/l13_3_evaluation.py
"""Lesson 13.3: DDPG, TD3 and SAC against PPO, and how to evaluate any agent (Level 13 checkpoint).

INPUT   mjcourse.envs.ReachEnv (joint_delta actions); mjcourse.rl.offpolicy and mjcourse.rl.ppo
PROCESS (1) DDPG, TD3, SAC (initial temperature 0.1) and SAC (initial temperature 1.0),
            40 000 environment steps, 3 seeds each in parallel: deterministic success on 20
            held-out initial states every 10 000 steps, then on 100;
        (2) PPO, 3 seeds, the same held-out evaluation every 10 000 steps up to 100 000;
        (3) the checkpoint: PPO with and without observation normalization, 5 seeds each,
            300 000 steps; per seed: held-out success on 100 states, mean steps to reach,
            environment steps until held-out success first reaches 90%; across seeds: mean,
            IQM and 95% bootstrap intervals;
        (4) evaluation pitfalls on the checkpoint runs: training success against held-out
            success; how the interval narrows with seeds; pooling episodes across seeds
OUTPUT  printed tables

Run:  python examples/l13_3_evaluation.py          (about fifteen minutes on 17 CPU cores)
"""
# requires: torch

import os
from multiprocessing import Pool

import numpy as np
import torch

from mjcourse import stats
from mjcourse.envs import ReachEnv
from mjcourse.rl import offpolicy, ppo

FAST = os.environ.get("MJC_FAST") == "1"
CURVE_STATES, FINAL_STATES = range(10_000, 10_020), range(10_000, 10_100)   # reset seeds never used in training


def make_env():
    return ReachEnv(action_mode="joint_delta")


def held_out(act, seeds) -> dict:
    """Deterministic episodes from the given reset seeds with `act(obs) -> action`."""
    env = make_env()
    success, length = [], []
    for seed in seeds:
        obs, _ = env.reset(seed=int(seed))
        steps, done = 0, False
        while not done:
            obs, _, terminated, truncated, info = env.step(np.asarray(act(obs), np.float32))
            steps, done = steps + 1, terminated or truncated
        success.append(info["is_success"])
        length.append(steps)
    env.close()
    return {"success": float(np.mean(success)), "length": float(np.mean(length))}


# ---------------------------------------------------------------- (1, 2) off-policy against PPO

def off_policy_run(job: tuple[str, float, int, int]) -> tuple[str, int, list[dict], dict]:
    algo, alpha, seed, steps = job
    torch.set_num_threads(1)
    cfg = offpolicy.OffPolicyConfig(algo=algo, init_alpha=alpha, total_steps=steps, seed=seed, eval_every=steps // 4)
    learner, history = offpolicy.train(make_env, cfg, lambda l: held_out(lambda o: l.act(o, explore=False), CURVE_STATES), log=None)
    final = held_out(lambda o: learner.act(o, explore=False), FINAL_STATES)
    name = f"{algo.upper()}" + (f" (temperature {alpha:g})" if algo == "sac" else "")
    return name, seed, history, final


def part1_2() -> None:
    steps, seeds = (2_000, 1) if FAST else (40_000, 3)
    jobs = [(algo, alpha, seed, steps) for algo, alpha in (("ddpg", 0.1), ("td3", 0.1), ("sac", 0.1), ("sac", 1.0))
            for seed in range(seeds)]
    if FAST:
        results = [off_policy_run(j) for j in jobs]
    else:
        with Pool(len(jobs)) as pool:
            results = pool.map(off_policy_run, jobs)
    marks = [h["steps"] for h in results[0][2]]
    print(f"  held-out success (20 states) at {', '.join(f'{m // 1000}k' for m in marks)} steps, then on 100 states; per seed")
    for name in dict.fromkeys(r[0] for r in results):
        rows = [r for r in results if r[0] == name]
        curves = "; ".join("/".join(f"{h['success']:.2f}" for h in r[2]) for r in rows)
        finals = ", ".join(f"{r[3]['success']:.2f}" for r in rows)
        print(f"  {name:<26} {curves:<44} final {finals}   ({rows[0][2][-1]['seconds']:.0f} s each)")
    ppo_steps = 10_000 if FAST else 100_000
    for seed in range(seeds):
        cfg = ppo.PPOConfig(total_steps=ppo_steps, seed=seed)
        agent, history = ppo.train(make_env, cfg, log=None, evaluate_fn=lambda a: held_out(a.act, CURVE_STATES), eval_every=10_000)
        evals = [(h["steps"], h["eval"]["success"]) for h in history if "eval" in h]
        print(f"  PPO seed {seed} (16 environments): " + ", ".join(f"{s // 1000}k {v:.2f}" for s, v in evals)
              + f"; final on 100 states {held_out(agent.act, FINAL_STATES)['success']:.2f} ({history[-1]['seconds']:.0f} s)")


# ---------------------------------------------------------------- (3, 4) the checkpoint

def checkpoint_run(normalize: bool, seed: int, steps: int) -> dict:
    cfg = ppo.PPOConfig(total_steps=steps, seed=seed, normalize_obs=normalize)
    agent, history = ppo.train(make_env, cfg, log=None, evaluate_fn=lambda a: held_out(a.act, CURVE_STATES), eval_every=10_000)
    final = held_out(agent.act, FINAL_STATES)
    reached = [h["steps"] for h in history if "eval" in h and h["eval"]["success"] >= 0.9]
    return {"success": final["success"], "length": final["length"], "steps_to_90": reached[0] if reached else np.nan,
            "train_success": history[-1]["success"], "seconds": history[-1]["seconds"]}


def summary(values) -> str:
    values = np.asarray(values, float)
    if np.isnan(values).any():
        return f"{np.isnan(values).sum()} seeds never reached it"
    mean, lo, hi = stats.bootstrap_ci(values)
    return f"mean {mean:.3g} [{lo:.3g}, {hi:.3g}], IQM {stats.iqm(values):.3g}"


def part3_4() -> None:
    steps, seeds = (10_000, 2) if FAST else (300_000, 5)
    runs = {}
    for normalize in (True, False):
        label = "normalized observations" if normalize else "raw observations"
        runs[label] = [checkpoint_run(normalize, seed, steps) for seed in range(seeds)]
        r = runs[label]
        print(f"  {label}: {steps // 1000}k steps per seed, {r[0]['seconds']:.0f} s each")
        print(f"    held-out success per seed  {', '.join(format(x['success'], '.2f') for x in r)}   {summary([x['success'] for x in r])}")
        print(f"    mean steps to reach        {', '.join(format(x['length'], '.1f') for x in r)}   {summary([x['length'] for x in r])}")
        print(f"    steps until 90% held-out   {', '.join('never' if np.isnan(x['steps_to_90']) else format(x['steps_to_90'] / 1000, '.0f') + 'k' for x in r)}   "
              f"{summary([x['steps_to_90'] / 1000 for x in r])} (thousands)")
    a, b = runs["normalized observations"], runs["raw observations"]
    for key, name in (("length", "mean steps to reach"), ("steps_to_90", "steps until 90%")):
        if not any(np.isnan([x[key] for x in a + b])):
            diff, lo, hi = stats.paired_difference_ci([x[key] for x in b], [x[key] for x in a])
            print(f"  raw minus normalized, {name}, matched by seed: {diff:.3g} [{lo:.3g}, {hi:.3g}]")
    print("(4) evaluation pitfalls, normalized runs")
    r = a
    print(f"  training success (stochastic policy, last 100 training episodes) per seed: "
          f"{', '.join(format(x['train_success'], '.2f') for x in r)}; held-out deterministic: {', '.join(format(x['success'], '.2f') for x in r)}")
    lengths = [x["length"] for x in r]
    for k in range(1, len(lengths) + 1):
        if k == 1:
            print(f"  mean steps to reach from 1 seed: {lengths[0]:.2f}, no interval possible")
        else:
            mean, lo, hi = stats.bootstrap_ci(lengths[:k])
            print(f"  mean steps to reach from {k} seeds: {mean:.2f} [{lo:.2f}, {hi:.2f}], width {hi - lo:.2f}")


if __name__ == "__main__":
    print(f"torch {torch.__version__}; ReachEnv joint_delta; held-out reset seeds 10000-10099")
    print("(1, 2) off-policy and PPO, deterministic evaluation on held-out states")
    part1_2()
    print("(3) checkpoint: PPO with and without observation normalization")
    part3_4()

Output:

torch 2.14.1+cpu; ReachEnv joint_delta; held-out reset seeds 10000-10099
(1, 2) off-policy and PPO, deterministic evaluation on held-out states
  held-out success (20 states) at 10k, 20k, 30k, 40k steps, then on 100 states; per seed
  DDPG                       0.05/0.35/0.20/0.80; 0.15/0.85/0.55/0.95; 0.00/0.45/0.65/0.75 final 0.77, 0.96, 0.76   (227 s each)
  TD3                        0.10/0.25/0.25/0.60; 0.00/0.10/0.45/0.50; 0.10/0.15/0.15/0.55 final 0.57, 0.44, 0.51   (246 s each)
  SAC (temperature 0.1)      0.05/0.80/1.00/1.00; 0.10/0.65/1.00/1.00; 0.10/0.80/1.00/1.00 final 1.00, 1.00, 1.00   (363 s each)
  SAC (temperature 1)        0.00/0.05/0.10/0.25; 0.00/0.00/0.20/0.10; 0.05/0.00/0.10/0.20 final 0.21, 0.15, 0.11   (374 s each)
  PPO seed 0 (16 environments): 10k 0.05, 20k 0.15, 30k 0.25, 40k 0.55, 51k 0.85, 61k 1.00, 71k 1.00, 81k 1.00, 90k 1.00; final on 100 states 1.00 (19 s)
  PPO seed 1 (16 environments): 10k 0.00, 20k 0.20, 30k 0.75, 40k 0.95, 51k 0.95, 61k 1.00, 71k 1.00, 81k 1.00, 90k 1.00; final on 100 states 1.00 (18 s)
  PPO seed 2 (16 environments): 10k 0.00, 20k 0.15, 30k 0.40, 40k 0.65, 51k 0.95, 61k 0.95, 71k 1.00, 81k 1.00, 90k 1.00; final on 100 states 1.00 (19 s)
(3) checkpoint: PPO with and without observation normalization
  normalized observations: 300k steps per seed, 52 s each
    held-out success per seed  1.00, 1.00, 1.00, 1.00, 1.00   mean 1 [1, 1], IQM 1
    mean steps to reach        6.8, 6.9, 7.0, 7.0, 6.9   mean 6.94 [6.87, 7.01], IQM 6.94
    steps until 90% held-out   61k, 41k, 51k, 61k, 51k   mean 53.2 [47.1, 59.4], IQM 54.6 (thousands)
  raw observations: 300k steps per seed, 52 s each
    held-out success per seed  1.00, 1.00, 1.00, 1.00, 1.00   mean 1 [1, 1], IQM 1
    mean steps to reach        7.2, 7.3, 7.5, 7.2, 7.5   mean 7.36 [7.23, 7.5], IQM 7.36
    steps until 90% held-out   61k, 61k, 72k, 72k, 72k   mean 67.6 [63.5, 71.7], IQM 68.3 (thousands)
  raw minus normalized, mean steps to reach, matched by seed: 0.424 [0.308, 0.532]
  raw minus normalized, steps until 90%, matched by seed: 1.43e+04 [6.14e+03, 2.05e+04]
(4) evaluation pitfalls, normalized runs
  training success (stochastic policy, last 100 training episodes) per seed: 1.00, 1.00, 1.00, 1.00, 1.00; held-out deterministic: 1.00, 1.00, 1.00, 1.00, 1.00
  mean steps to reach from 1 seed: 6.82, no interval possible
  mean steps to reach from 2 seeds: 6.86 [6.82, 6.90], width 0.08
  mean steps to reach from 3 seeds: 6.92 [6.82, 7.05], width 0.23
  mean steps to reach from 4 seeds: 6.94 [6.86, 7.02], width 0.16
  mean steps to reach from 5 seeds: 6.94 [6.87, 7.01], width 0.14

Debugging

SAC learns nothing for tens of thousands of steps. The entropy weight dwarfs the reward. Compare $\alpha$ times the action dimension with the typical reward per step; start $\alpha$ lower or rescale the reward.

The critic’s values grow without bound. Overestimation feeding on itself: use the minimum of two critics, lower the learning rate, or check that terminal transitions really cut the bootstrap.

Off-policy training is very slow by the clock. One gradient step per environment step on a CPU. Batch environment steps, take several gradient steps per batch, or use a GPU for the networks.

Two methods have overlapping intervals and the paper says one is better. Check how many seeds, how the interval was computed, and whether the metric saturated.

Exercise

Run TD3 for 120 000 steps with 3 seeds and report whether it catches DDPG. Then log the mean critic value on a fixed batch of states for both, every 10 000 steps, and say whether DDPG’s critic overestimates here.

Challenge

Repeat the checkpoint on the reach-and-hold variant of Lesson 13.2, where success does not saturate as quickly, with the final distance at 5 s as the primary metric. Report the paired difference with a bootstrap interval and an interquartile mean over at least 5 seeds, and state in one sentence what the experiment can and cannot support.

Research connection

[!research] How results are reported decides what is believed Henderson et al. (arXiv:1709.06560) showed that seeds, implementations and hyperparameters can change the ranking of deep RL methods on MuJoCo tasks, and Agarwal et al. (arXiv:2108.13264) proposed interquartile means and bootstrap intervals over runs because point estimates from a few seeds mislead. Both are in the reading list. Their recommendations apply directly to robot-learning claims (inference): a method evaluated on in-distribution initial states, with one seed, on a metric that saturates, has not been compared with anything.

{"id": "13.3-check", "title": "Knowledge check", "questions": [
  {"kind": "mcq", "q": "Which addition is specific to TD3, not SAC?",
   "options": ["Two critics with the minimum in the target", "Noise on the target action and delayed actor updates", "A replay buffer", "Target networks"],
   "answer": 1,
   "explain": "<p>Both use two critics; target policy smoothing and delayed actor updates are TD3's.</p>"},
  {"kind": "mcq", "q": "SAC barely learns with an initial temperature of 1.0 but learns quickly with 0.1. What is the most likely reason?",
   "options": ["The replay buffer is too small", "The entropy bonus is much larger than the per-step reward, so the policy is rewarded mostly for randomness", "The critic overestimates", "The learning rate is too high"],
   "answer": 1,
   "explain": "<p>Output (1): rewards of about −0.3 per step against an entropy bonus scaled by alpha = 1 over a 7-dimensional action.</p>"},
  {"kind": "mcq", "q": "Two configurations both reach 100% held-out success in every seed. What should you report?",
   "options": ["That they are equivalent", "A metric that does not saturate, such as time to a threshold or final error, with paired intervals", "The training success instead", "Only the best seed"],
   "answer": 1,
   "explain": "<p>Output (3): success saturates; steps to 90% and steps to reach still separate them.</p>"},
  {"kind": "mcq", "q": "A bootstrap interval over 2 seeds is narrower than one over 5 seeds of the same experiment. Why?",
   "options": ["Two seeds are more reliable", "With two values the bootstrap has almost no variation to resample, so it understates uncertainty", "The bootstrap is wrong for means", "The fifth seed is an outlier"],
   "answer": 1,
   "explain": "<p>Output (4): width 0.08 from 2 seeds against 0.14 from 5.</p>"}
]}

Next

Level 14 learns from demonstrations instead of rewards: behaviour cloning, the covariate shift that undoes it, and DAgger.