Proximal Policy Optimization (PPO) improves a stochastic policy from its own recent experience. Each iteration runs the current policy in several environments, estimates how much better each action was than the policy’s average there (the advantage of Lesson 13.1) using a learned value function, and then takes several gradient steps that make good actions more likely, while a clip on the probability ratio keeps the new policy close to the one that collected the data. Everything in this lesson is in mjcourse/rl/ppo.py, about 200 lines, and it learns to move a seven-joint arm to a target in under a minute.
The implementation is short; what takes skill is reading it while it trains. PPO logs five numbers alongside the return (approximate KL, clip fraction, entropy, value loss, explained variance), and each has a healthy range. This lesson trains it once with defaults, then twice with settings that break it, and shows what each breakage looks like in those numbers before it shows in the result.
[!established] The clipped objective With the probability ratio $\rho_t(\theta) = \pi_\theta(a_t\mid s_t)/\pi_{\theta_\text{old}}(a_t\mid s_t)$ and advantage estimates $\hat A_t$, PPO maximizes $\mathbb{E}t[\min(\rho_t \hat A_t,\ \text{clip}(\rho_t, 1-\epsilon, 1+\epsilon)\hat A_t)]$, so moving the ratio beyond $1 \pm \epsilon$ earns nothing (Schulman et al., arXiv:1707.06347). The advantages come from generalized advantage estimation, $\hat A_t = \sum_k (\gamma\lambda)^k \delta{t+k}$ with $\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$ (Schulman et al., arXiv:1506.02438). Both papers are in the reading list.
This lesson has no lab: training PPO takes 16 processes and a minute, more than a browser tab should spend. Its picture is output (1)’s learning curve. Within 60 000 environment steps (about 10 s) the arm goes from reaching 16% of targets to 95%. The episode length then keeps falling, from 27 steps to 7, as the policy learns to reach faster. All the while the policy’s standard deviation shrinks from 0.60 to 0.45: the policy becomes less random as it becomes confident.
info["final_obs"]. Terminated episodes get nothing added.| Signal | Healthy here | What it measures |
|---|---|---|
| approximate KL | ≈ 0.01 | how far each update moved the policy, $\mathbb{E}[(\rho - 1) - \log\rho]$ |
| clip fraction | 0.10 to 0.20 | share of samples whose ratio left $[0.8, 1.2]$ |
| entropy, policy std | falling slowly | how random the policy still is |
| value loss | falling | error of $V$ on the returns it is fit to |
| explained variance | rising toward 1 | $1 - \text{Var}(R - V)/\text{Var}(R)$: how much of the return $V$ predicts |
Output (1) stays in those ranges throughout: KL 0.007 to 0.013, clip fraction 0.10 to 0.19, standard deviation falling from 0.60 to 0.45. Explained variance dips from 0.80 to 0.39 while the policy changes fastest (around 60 000 steps, when the returns it predicts are shifting) and then rises to 0.95. The held-out evaluation, deterministic on 100 initial states never used in training, reaches every target (median final distance 13.9 mm; episodes end when the distance falls below 20 mm).
Output (2) breaks it twice, one seed each, as illustrations:
Both runs end with held-out success of 99 to 100%, which is the point: the diagnostics flagged them long before the result did. A KL above about 0.05 or a clip fraction above about 0.3 here means the step size, not the task, is in charge (inference from these runs; the thresholds depend on the problem).
Output (3) changes the task so that time limits matter: the arm must reach and hold, the episode never terminates, and every episode is truncated at 5 s. Correctly bootstrapped, three seeds hold the target with median errors of 2.3, 1.7 and 1.6 mm on the held-out states; with truncation treated as terminal (the mistake that passes check_env in Lesson 12.1), 4.4, 2.3 and 2.6 mm, with worse 90th percentiles (10.1, 4.4, 5.0 mm against 3.8, 2.6, 2.6 mm) and worse returns in every seed. Treating the time limit as terminal tells the value function that the last states of every episode have no future, which is false for a hold task. With three seeds per arm the smallest attainable two-sided Mann-Whitney p is 0.1, so “worse in every seed” is as strong a statement as this experiment supports.
INPUT: ReachEnv with joint-target actions; mjcourse.rl.ppo
PROCESS: train with defaults; train with a large learning rate and with many epochs; train reach-and-hold with and without bootstrapping truncation, 3 seeds each
OUTPUT: learning curves with diagnostics, held-out evaluations
```python file=src/mjcourse/rl/ppo.py “"”Proximal Policy Optimization, written out (Lesson 13.2).
One iteration: run the current policy in N vector environments for T steps; estimate
advantages with GAE(lambda) from a learned value function; then take several epochs of
minibatch gradient steps on the clipped surrogate objective, a value loss and an entropy
bonus. Truncated episodes are bootstrapped from the value of their final observation
(gymnasium’s SAME_STEP autoreset puts it in info[“final_obs”]); terminated ones are not.
Observations are normalized by running statistics unless normalize_obs is False.
“””
from future import annotations
import time from dataclasses import dataclass, field
import gymnasium as gym import numpy as np import torch from torch import nn
@dataclass class PPOConfig: num_envs: int = 16 steps_per_env: int = 128 # T: rollout length per environment and iteration total_steps: int = 500_000 gamma: float = 0.99 gae_lambda: float = 0.95 epochs: int = 10 minibatch: int = 512 lr: float = 3e-4 clip: float = 0.2 entropy_coef: float = 0.0 value_coef: float = 0.5 max_grad_norm: float = 0.5 hidden: tuple[int, …] = (64, 64) normalize_obs: bool = True bootstrap_truncation: bool = True # False treats time limits as terminal states (a mistake, kept to measure it) seed: int = 0 log_every: int = 10 extra: dict = field(default_factory=dict)
class RunningMeanStd: “"”Running mean and variance (parallel algorithm of Chan et al.) for observation normalization.”””
def __init__(self, shape):
self.mean, self.var, self.count = np.zeros(shape), np.ones(shape), 1e-4
def update(self, x: np.ndarray) -> None:
batch_mean, batch_var, n = x.mean(axis=0), x.var(axis=0), x.shape[0]
delta, total = batch_mean - self.mean, self.count + n
self.mean = self.mean + delta * n / total
self.var = (self.var * self.count + batch_var * n + delta**2 * self.count * n / total) / total
self.count = total
def __call__(self, x: np.ndarray) -> np.ndarray:
return np.clip((x - self.mean) / np.sqrt(self.var + 1e-8), -10, 10)
def mlp(sizes, out_std: float) -> nn.Sequential: layers = [] for a, b in zip(sizes[:-2], sizes[1:-1]): layers += [nn.Linear(a, b), nn.Tanh()] layers.append(nn.Linear(sizes[-2], sizes[-1])) for layer in layers: if isinstance(layer, nn.Linear): nn.init.orthogonal_(layer.weight, np.sqrt(2)) nn.init.zeros_(layer.bias) nn.init.orthogonal_(layers[-1].weight, out_std) return nn.Sequential(*layers)
class ActorCritic(nn.Module): “"”Gaussian policy with a state-independent log standard deviation, and a separate value network.”””
def __init__(self, obs_dim: int, act_dim: int, hidden: tuple[int, ...]):
super().__init__()
self.actor = mlp((obs_dim, *hidden, act_dim), out_std=0.01)
self.critic = mlp((obs_dim, *hidden, 1), out_std=1.0)
self.log_std = nn.Parameter(torch.full((act_dim,), -0.5))
def dist(self, obs: torch.Tensor) -> torch.distributions.Normal:
return torch.distributions.Normal(self.actor(obs), self.log_std.exp())
def value(self, obs: torch.Tensor) -> torch.Tensor:
return self.critic(obs).squeeze(-1)
class Agent:
“"”The trained policy with its observation normalizer; act returns the mean action.”””
def __init__(self, net: ActorCritic, obs_rms: RunningMeanStd | None, low, high):
self.net, self.obs_rms, self.low, self.high = net, obs_rms, low, high
def norm(self, obs: np.ndarray) -> np.ndarray:
return self.obs_rms(obs) if self.obs_rms is not None else obs
@torch.no_grad()
def act(self, obs: np.ndarray) -> np.ndarray:
x = torch.as_tensor(self.norm(np.asarray(obs, np.float64)), dtype=torch.float32)
return np.clip(self.net.actor(x).numpy(), self.low, self.high)
def train(make_env, cfg: PPOConfig, log=print, evaluate_fn=None, eval_every: int = 0) -> tuple[Agent, list[dict]]:
“"”Returns the agent and one history row per iteration; with evaluate_fn, rows at every
eval_every environment steps also hold its result (deterministic held-out evaluation).”””
torch.manual_seed(cfg.seed)
envs = gym.vector.AsyncVectorEnv([make_env] * cfg.num_envs, autoreset_mode=gym.vector.AutoresetMode.SAME_STEP)
obs_dim, act_dim = envs.single_observation_space.shape[0], envs.single_action_space.shape[0]
low, high = envs.single_action_space.low, envs.single_action_space.high
net = ActorCritic(obs_dim, act_dim, cfg.hidden)
optimizer = torch.optim.Adam(net.parameters(), lr=cfg.lr, eps=1e-5)
obs_rms = RunningMeanStd(obs_dim) if cfg.normalize_obs else None
agent = Agent(net, obs_rms, low, high)
n, t_len = cfg.num_envs, cfg.steps_per_env
iterations = cfg.total_steps // (n * t_len)
obs, _ = envs.reset(seed=cfg.seed)
history, finished = [], {“return”: [], “success”: [], “length”: []}
ep_return, ep_length = np.zeros(n), np.zeros(n, int)
start = time.perf_counter()
for it in range(1, iterations + 1):
b_obs, b_act, b_logp = np.zeros((t_len, n, obs_dim)), np.zeros((t_len, n, act_dim)), np.zeros((t_len, n))
b_rew, b_done, b_val = np.zeros((t_len, n)), np.zeros((t_len, n)), np.zeros((t_len + 1, n))
for t in range(t_len):
if obs_rms is not None:
obs_rms.update(obs)
x = torch.as_tensor(agent.norm(obs), dtype=torch.float32)
with torch.no_grad():
dist, value = net.dist(x), net.value(x)
action = dist.sample()
b_obs[t], b_act[t], b_logp[t], b_val[t] = x.numpy(), action.numpy(), dist.log_prob(action).sum(-1).numpy(), value.numpy()
obs, reward, terminated, truncated, info = envs.step(np.clip(action.numpy(), low, high))
reward = reward.astype(np.float64)
ep_return += reward
ep_length += 1
for i in np.flatnonzero(truncated & ~terminated & cfg.bootstrap_truncation): # the episode did not end
final = torch.as_tensor(agent.norm(info[“final_obs”][i][None]), dtype=torch.float32)
with torch.no_grad():
reward[i] += cfg.gamma * net.value(final).item()
b_rew[t], b_done[t] = reward, terminated | truncated
for i in np.flatnonzero(terminated | truncated):
finished[“return”].append(ep_return[i])
finished[“length”].append(ep_length[i])
final_info = info.get(“final_info”, {}) # the finished episode’s info
finished[“success”].append(bool(final_info[“is_success”][i]) if “is_success” in final_info else False)
ep_return[i], ep_length[i] = 0.0, 0
with torch.no_grad():
b_val[t_len] = net.value(torch.as_tensor(agent.norm(obs), dtype=torch.float32)).numpy()
advantages, last = np.zeros((t_len, n)), np.zeros(n)
for t in reversed(range(t_len)): # GAE(lambda)
alive = 1.0 - b_done[t]
delta = b_rew[t] + cfg.gamma * b_val[t + 1] * alive - b_val[t]
last = delta + cfg.gamma * cfg.gae_lambda * alive * last
advantages[t] = last
returns = advantages + b_val[:t_len]
flat = lambda a: torch.as_tensor(a.reshape(t_len * n, *a.shape[2:]), dtype=torch.float32) # noqa: E731
obs_t, act_t, logp_t, adv_t, ret_t = flat(b_obs), flat(b_act), flat(b_logp), flat(advantages), flat(returns)
stats = {“kl”: [], “clipfrac”: [], “value_loss”: [], “entropy”: []}
for _ in range(cfg.epochs):
for idx in torch.randperm(t_len * n).split(cfg.minibatch):
dist = net.dist(obs_t[idx])
logp = dist.log_prob(act_t[idx]).sum(-1)
ratio = (logp - logp_t[idx]).exp()
adv = (adv_t[idx] - adv_t[idx].mean()) / (adv_t[idx].std() + 1e-8)
policy_loss = -torch.min(ratio * adv, ratio.clamp(1 - cfg.clip, 1 + cfg.clip) * adv).mean()
value_loss = 0.5 * (net.value(obs_t[idx]) - ret_t[idx]).pow(2).mean()
entropy = dist.entropy().sum(-1).mean()
loss = policy_loss + cfg.value_coef * value_loss - cfg.entropy_coef * entropy
optimizer.zero_grad()
loss.backward()
nn.utils.clip_grad_norm_(net.parameters(), cfg.max_grad_norm)
optimizer.step()
with torch.no_grad():
stats[“kl”].append(((ratio - 1) - (logp - logp_t[idx])).mean().item()) # low-variance KL estimate
stats[“clipfrac”].append(((ratio - 1).abs() > cfg.clip).float().mean().item())
stats[“value_loss”].append(value_loss.item())
stats[“entropy”].append(entropy.item())
variance = ret_t.var().item()
with torch.no_grad():
explained = 1 - (ret_t - net.value(obs_t)).var().item() / variance if variance > 0 else float(“nan”)
recent = slice(-100, None)
row = {“iteration”: it, “steps”: it * n * t_len, “seconds”: time.perf_counter() - start,
“return”: float(np.mean(finished[“return”][recent])) if finished[“return”] else float(“nan”),
“success”: float(np.mean(finished[“success”][recent])) if finished[“success”] else float(“nan”),
“length”: float(np.mean(finished[“length”][recent])) if finished[“length”] else float(“nan”),
“explained_variance”: explained, “std”: float(net.log_std.exp().mean().item()),
**{k: float(np.mean(v)) for k, v in stats.items()}}
if evaluate_fn is not None and eval_every and (it * n * t_len) // eval_every > ((it - 1) * n * t_len) // eval_every:
row[“eval”] = evaluate_fn(agent)
history.append(row)
if log and (it % cfg.log_every == 0 or it == iterations):
log(row)
envs.close()
return agent, history
def evaluate(agent: Agent, make_env, seeds) -> dict: “"”Deterministic (mean-action) episodes from the given reset seeds.””” env = make_env() success, final_distance, returns = [], [], [] for seed in seeds: obs, info = env.reset(seed=int(seed)) total, done = 0.0, False while not done: obs, reward, terminated, truncated, info = env.step(agent.act(obs)) total += reward done = terminated or truncated success.append(bool(info.get(“is_success”, False))) final_distance.append(info.get(“distance”, float(“nan”))) returns.append(total) env.close() return {“success”: np.array(success), “final_distance”: np.array(final_distance), “return”: np.array(returns)}
```python file=examples/l13_2_ppo.py
"""Lesson 13.2: PPO from scratch on a seven-joint arm, and how its diagnostics read.
INPUT mjcourse.envs.ReachEnv (joint_delta actions, 7 joints), mjcourse.rl.ppo
PROCESS (1) PPO with the default configuration, 300 000 steps: learning curve with the
diagnostics (approximate KL, clip fraction, entropy, value loss, explained
variance, policy standard deviation), then 100 held-out initial states;
(2) the same with two bad settings, learning rate x10 and 40 epochs per batch:
what each does to the diagnostics and to the result;
(3) reach and hold (no termination on success, every episode truncated at 5 s),
500 000 steps, with truncation bootstrapped and with it treated as terminal,
3 seeds each
OUTPUT printed tables
Run: python examples/l13_2_ppo.py (about ten minutes on 17 CPU cores)
"""
# requires: torch
import os
import numpy as np
import torch
from mjcourse.envs import ReachEnv
from mjcourse.rl.ppo import PPOConfig, evaluate, train
FAST = os.environ.get("MJC_FAST") == "1"
HELD_OUT = range(10_000, 10_100) # reset seeds never seen in training
COLUMNS = ("steps", "return", "success", "length", "kl", "clipfrac", "entropy", "value_loss", "explained_variance", "std")
HEADERS = ("steps", "return", "success", "length", "approx KL", "clip frac", "entropy", "value loss", "explained var", "policy std")
def show_curve(history: list[dict], every: int) -> None:
print(" " + "".join(f"{c:>14}" if c == "explained var" else f"{c:>12}" for c in HEADERS))
for row in history[every - 1::every]:
print(" " + "".join(f"{row[c]:>12.0f}" if c == "steps" else f"{row[c]:>14.3f}" if c == "explained_variance" else f"{row[c]:>12.3f}" for c in COLUMNS))
def run(label: str, make, cfg: PPOConfig, every: int | None) -> dict:
agent, history = train(make, cfg, log=None)
result = evaluate(agent, make, HELD_OUT if not FAST else range(10_000, 10_010))
print(f" {label}: {history[-1]['seconds']:.0f} s")
if every:
show_curve(history, every)
print(f" held-out: success {result['success'].mean():.0%}, final distance median "
f"{1000 * np.median(result['final_distance']):.1f} mm, 90th percentile {1000 * np.percentile(result['final_distance'], 90):.1f} mm, "
f"mean return {result['return'].mean():.2f}")
return result
if __name__ == "__main__":
torch.set_num_threads(1)
scale = 0.1 if FAST else 1.0
reach = lambda: ReachEnv(action_mode="joint_delta") # noqa: E731
hold = lambda: ReachEnv(action_mode="joint_delta", terminate_on_success=False) # noqa: E731
base = dict(num_envs=16, steps_per_env=128, seed=0)
print(f"torch {torch.__version__}; 16 environments x 128 steps = 2048 transitions per iteration")
print("(1) default configuration: lr 3e-4, 10 epochs, minibatch 512, clip 0.2, gamma 0.99, lambda 0.95")
run("reach, default", reach, PPOConfig(total_steps=int(300_000 * scale), **base), every=15)
print("(2) bad settings")
run("reach, learning rate 3e-3", reach, PPOConfig(total_steps=int(300_000 * scale), lr=3e-3, **base), every=15)
run("reach, 40 epochs per batch", reach, PPOConfig(total_steps=int(300_000 * scale), epochs=40, **base), every=15)
print("(3) reach and hold: every episode is truncated at 100 steps (5 s); curves for seed 0")
medians = {}
for label, boot in (("truncation bootstrapped", True), ("truncation treated as terminal", False)):
medians[label] = []
for seed in range(1 if FAST else 3):
cfg = PPOConfig(total_steps=int(500_000 * scale), bootstrap_truncation=boot, **{**base, "seed": seed})
result = run(f"hold, {label}, seed {seed}", hold, cfg, every=40 if seed == 0 else None)
medians[label].append(1000 * np.median(result["final_distance"]))
for label, values in medians.items():
print(f" {label}: held-out median final distance per seed {', '.join(f'{v:.1f}' for v in values)} mm")
Output:
torch 2.14.1+cpu; 16 environments x 128 steps = 2048 transitions per iteration
(1) default configuration: lr 3e-4, 10 epochs, minibatch 512, clip 0.2, gamma 0.99, lambda 0.95
reach, default: 46 s
steps return success length approx KL clip frac entropy value loss explained var policy std
30720 -12.262 0.160 90.950 0.007 0.099 6.396 0.540 0.801 0.604
61440 -2.012 0.950 26.870 0.011 0.154 6.217 0.179 0.392 0.588
92160 -1.103 1.000 11.790 0.010 0.151 5.890 0.025 0.740 0.561
122880 -0.967 1.000 9.630 0.013 0.185 5.604 0.016 0.807 0.540
153600 -0.987 1.000 8.970 0.009 0.132 5.300 0.010 0.865 0.516
184320 -0.929 1.000 8.340 0.010 0.143 5.108 0.008 0.902 0.504
215040 -0.796 1.000 7.280 0.010 0.153 4.823 0.007 0.901 0.485
245760 -0.893 1.000 7.840 0.009 0.132 4.619 0.004 0.936 0.471
276480 -0.793 1.000 7.180 0.009 0.135 4.282 0.004 0.949 0.449
held-out: success 100%, final distance median 13.9 mm, 90th percentile 18.3 mm, mean return -0.83
(2) bad settings
reach, learning rate 3e-3: 40 s
steps return success length approx KL clip frac entropy value loss explained var policy std
30720 -11.554 0.260 84.300 0.041 0.416 5.953 0.447 0.909 0.566
61440 -2.919 0.860 36.750 0.055 0.489 5.400 0.262 0.944 0.523
92160 -1.377 1.000 15.140 0.067 0.524 4.481 0.015 0.917 0.459
122880 -1.094 1.000 10.580 0.080 0.546 3.466 0.005 0.953 0.398
153600 -0.952 1.000 9.440 0.140 0.610 2.395 0.004 0.949 0.342
184320 -0.971 1.000 8.250 0.138 0.635 1.317 0.002 0.975 0.293
215040 -0.864 1.000 7.710 0.563 0.644 0.387 0.011 0.982 0.256
245760 -0.908 1.000 7.650 0.534 0.690 -0.505 0.004 0.970 0.226
276480 -0.926 0.990 8.780 1.034 0.699 -1.558 0.032 0.994 0.194
held-out: success 100%, final distance median 16.3 mm, 90th percentile 19.1 mm, mean return -0.86
reach, 40 epochs per batch: 64 s
steps return success length approx KL clip frac entropy value loss explained var policy std
30720 -10.578 0.260 84.950 0.017 0.263 5.782 0.501 0.879 0.552
61440 -2.195 0.950 26.950 0.016 0.262 4.976 0.106 0.973 0.492
92160 -1.386 1.000 13.550 0.017 0.270 4.056 0.008 0.948 0.431
122880 -1.108 1.000 10.580 0.019 0.310 2.988 0.003 0.975 0.370
153600 -0.902 1.000 8.330 0.023 0.356 1.817 0.001 0.986 0.315
184320 -0.958 1.000 7.820 0.041 0.327 0.695 0.002 0.989 0.267
215040 -0.864 1.000 7.480 0.028 0.378 -0.542 0.001 0.991 0.226
245760 -0.857 1.000 7.130 0.032 0.409 -1.710 0.001 0.989 0.191
276480 -0.845 0.990 7.870 0.106 0.354 -2.868 0.003 0.994 0.161
held-out: success 99%, final distance median 14.5 mm, 90th percentile 19.0 mm, mean return -0.91
(3) reach and hold: every episode is truncated at 100 steps (5 s); curves for seed 0
hold, truncation bootstrapped, seed 0: 63 s
steps return success length approx KL clip frac entropy value loss explained var policy std
81920 -5.185 0.040 100.000 0.008 0.114 6.338 0.040 0.914 0.599
163840 -2.574 0.640 100.000 0.011 0.148 5.717 0.005 0.931 0.548
245760 -1.779 0.980 100.000 0.010 0.144 5.044 0.001 0.919 0.498
327680 -1.417 1.000 100.000 0.010 0.146 4.025 0.001 0.951 0.434
409600 -1.351 1.000 100.000 0.013 0.191 3.144 0.000 0.965 0.385
491520 -1.219 1.000 100.000 0.014 0.193 2.216 0.000 0.971 0.339
held-out: success 100%, final distance median 2.3 mm, 90th percentile 3.8 mm, mean return -1.08
hold, truncation bootstrapped, seed 1: 61 s
held-out: success 100%, final distance median 1.7 mm, 90th percentile 2.6 mm, mean return -1.06
hold, truncation bootstrapped, seed 2: 61 s
held-out: success 100%, final distance median 1.6 mm, 90th percentile 2.6 mm, mean return -1.05
hold, truncation treated as terminal, seed 0: 63 s
steps return success length approx KL clip frac entropy value loss explained var policy std
81920 -6.156 0.050 100.000 0.009 0.141 6.128 0.128 0.785 0.582
163840 -3.180 0.220 100.000 0.010 0.146 5.422 0.012 0.875 0.524
245760 -2.394 0.590 100.000 0.010 0.150 4.933 0.004 0.913 0.490
327680 -1.760 0.940 100.000 0.013 0.194 4.160 0.002 0.941 0.440
409600 -1.557 0.960 100.000 0.014 0.214 3.264 0.001 0.952 0.389
491520 -1.330 1.000 100.000 0.012 0.174 2.348 0.001 0.975 0.343
held-out: success 99%, final distance median 4.4 mm, 90th percentile 10.1 mm, mean return -1.20
hold, truncation treated as terminal, seed 1: 62 s
held-out: success 100%, final distance median 2.3 mm, 90th percentile 4.4 mm, mean return -1.10
hold, truncation treated as terminal, seed 2: 65 s
held-out: success 100%, final distance median 2.6 mm, 90th percentile 5.0 mm, mean return -1.14
truncation bootstrapped: held-out median final distance per seed 2.3, 1.7, 1.6 mm
truncation treated as terminal: held-out median final distance per seed 4.4, 2.3, 2.6 mm
The return improves, then collapses. Steps too large: check KL and clip fraction first. Lower the learning rate or the epochs, or stop each update early when the KL passes a threshold.
Explained variance stays near zero. The value function predicts nothing, so advantages are noise. Check observation normalization, the value loss scale, and whether rewards are large or badly scaled.
The policy’s standard deviation collapses early. Over-fitting each batch, or an entropy bonus of zero on a task that needs exploration. Add a small entropy coefficient and watch the curve.
A hold or balance task never becomes precise. Truncation treated as termination. Print the terminated and truncated flags at the time limit.
Success during training is high and held-out success is low. Training success is measured with the stochastic policy on training states. Evaluate the mean action on states the agent never saw (Lesson 13.3).
Train the default configuration with the entropy coefficient at 0.01 and at 0.1, one seed each, and describe what changes in the entropy, the standard deviation, and the time to 95% training success.
Add early stopping per update: stop the epochs when the mean approximate KL of the last minibatch exceeds 0.02. Re-run the learning-rate-3e-3 configuration with it on 3 seeds and report whether the diagnostics return to the healthy ranges and what the cost is in environment steps to 95% success.
[!research] Implementation details are part of the algorithm The PPO of this lesson includes choices the original paper does not describe in full: orthogonal initialization, a state-independent standard deviation, observation normalization, advantage normalization per minibatch, gradient clipping and Adam’s epsilon. Engstrom et al. (arXiv:2005.12729) found that such details account for much of PPO’s advantage over TRPO in their experiments, and Andrychowicz et al. (arXiv:2006.05990) measured dozens of them at scale. A paper that reports “PPO” without its configuration has not specified its method; the configuration dataclass at the top of
ppo.pyis the minimum to publish with a result.
{"id": "13.2-check", "title": "Knowledge check", "questions": [
{"kind": "mcq", "q": "During training the clip fraction rises to 0.6 and the approximate KL to 0.5 while the return still improves. What is happening?",
"options": ["Training is healthy", "Updates are too large: most samples are clipped and the trust region no longer holds", "The value function is perfect", "The environment is deterministic"],
"answer": 1,
"explain": "<p>Output (2), learning rate 3e-3: KL up to 1.03, clip fraction 0.70, and a collapsing policy spread.</p>"},
{"kind": "numeric", "q": "With gamma = 0.99 and lambda = 0.95, by what factor does GAE weight the TD error 10 steps ahead relative to the current one?",
"answer": 0.541, "tol": 0.003,
"explain": "<p>$(\\gamma\\lambda)^{10} = 0.9405^{10}$ = 0.541.</p>"},
{"kind": "mcq", "q": "Explained variance is 0.95. What does that say?",
"options": ["The policy reaches 95% of targets", "The value function predicts 95% of the variance of the returns it is trained on", "95% of samples were clipped", "The entropy is 95% of its initial value"],
"answer": 1,
"explain": "<p>$1 - \\text{Var}(R - V)/\\text{Var}(R)$: a good value function makes advantage estimates informative.</p>"},
{"kind": "mcq", "q": "In a reach-and-hold task every episode is truncated at the time limit. What happens if truncation is treated as termination?",
"options": ["Nothing; only terminations matter", "The value function learns that the last states have no future, and the hold becomes less precise", "The episode length doubles", "PPO cannot run"],
"answer": 1,
"explain": "<p>Output (3): worse median and 90th-percentile errors in every seed.</p>"}
]}
Lesson 13.3 adds the off-policy family (DDPG, TD3, SAC) and the part every comparison depends on: evaluating agents on held-out states with enough seeds, and reporting the result with an interval.