Reinforcement learning names the parts of a loop you already run. The state is what the simulator needs to continue: qpos, qvel, act, plus anything the task added (here the target). The observation is what the policy is given, which may be less. The action is what the policy writes into ctrl, held for some number of physics steps. The reward is a number computed from the state after the step, the discount says how much later rewards count, and an episode ends by termination or truncation (Lesson 12.1). An algorithm’s job is to change the policy so that the expected discounted return goes up.
The simplest algorithm that does that treats the policy’s parameters as a vector to optimize by sampling: try a population of parameter vectors, keep the best, refit, repeat. That is the cross-entropy method, and on a small policy it is fast enough to watch. This lesson runs it on the two-link arm of Level 5, then uses the result to make the value function and the action-value function concrete.
[!established] Values For a policy $\pi$ and discount $\gamma$, the value of a state is the expected discounted return from it, $V^\pi(s) = \mathbb{E}\pi[\sum_k \gamma^k r_k \mid s_0 = s]$, and the action value is the same with the first action fixed, $Q^\pi(s, a) = \mathbb{E}\pi[\sum_k \gamma^k r_k \mid s_0 = s, a_0 = a]$. They are related by $V^\pi(s) = \mathbb{E}_{a \sim \pi(s)}[Q^\pi(s, a)]$, and the advantage $A^\pi(s,a) = Q^\pi(s,a) - V^\pi(s)$ says how much better than the policy’s average an action is. These are the standard definitions (R. S. Sutton, A. G. Barto, Reinforcement Learning: An Introduction, 2nd edition, 2018, free online).
Start search runs the cross-entropy method on the arm’s policy while the arm plays the current mean policy, moving to a new random target every 2 s.
```lab rl {“dock”: true, “title”: “A policy search you can watch”, “height”: 280}
## Mathematics
### The task as an MDP
Output (1) spells it out. The arm starts at its `bent` keyframe; a target is drawn in its plane (0.35 to 0.85 m from the shoulder). The policy observes the error $e = $ target $-$ ee in the x-z plane and the joint velocities, and outputs two torques held for 20 ms (10 physics steps). Each control step earns $r = -\|e\|$ in metres, and an episode is 100 control steps (2 s). With $\gamma = 0.99$ the effective horizon $1/(1-\gamma)$ is 100 steps, the whole episode.
The policy is linear, $u = \text{clip}(W_e e + W_v \dot q + b)$: 10 numbers, scaled so that the search works in units near 1. It does not see $q$, so it cannot know the gravity torque, which depends on $q$: the observation is not a Markov state for this policy. Its equilibrium satisfies $W_e e = g(q) - b$, so it stops a distance of roughly gravity torque over gain from the target. The second policy class adds $W_g\,(\cos q_1, \cos(q_1+q_2))$. The arm's gravity torques are exactly linear in those two features, so this class contains exact gravity compensation and can drive the equilibrium error to zero.
### The cross-entropy method
Each iteration samples 32 parameter vectors from a Gaussian $\mathcal{N}(\mu, \text{diag}\,\sigma^2)$, scores each by its mean return on 4 random targets shared by the whole population (shared targets make the comparison fair), takes the best 6 and sets $\mu$, $\sigma$ to their mean and standard deviation (plus a floor of 0.01 so the search never freezes). Nothing is differentiated: the method needs only returns, which is why it works on any simulator, and why it scales poorly with the number of parameters.
Output (2) runs it for 60 iterations with 5 seeds per policy class, in parallel, and evaluates each final mean policy on 100 held-out targets never used in the search:
| Policy | Success per seed (final distance < 2 cm) | Median final distance per seed |
|---|---|---|
| observes $e$, $\dot q$ | 29, 59, 76, 87, 64% | 31, 18, 14, 12, 16 mm |
| also $\cos q_1, \cos(q_1+q_2)$ | 83, 88, 45, 93, 85% | 7, 6, 21, 7, 8 mm |
The physics says the second class should win, and in 4 of 5 seeds it does: median success 85% against 64%, median distance 7 mm against 16 mm. With 5 seeds per class neither difference is significant (Mann-Whitney, p = 0.22 and p = 0.095), because one seed of each class landed far from the others. Seed-to-seed variation of a search this size is as large as the effect of the observation, and a single run of each would have supported either conclusion. Level 13.3 returns to how many seeds an RL comparison needs.
### Values, measured
Output (3) takes the policy of seed 0, adds Gaussian noise ($\sigma$ = 1 N m) to its torques so that it is stochastic, and estimates by 300 rollouts each the value of the start state with a target 8.5 cm away and the action values of six first actions:
| First action | $Q(s_0, a)$ | $A = Q - V$ |
|---|---|---|
| policy mean (16.5, 9.9) N m | −1.661 | −0.002 |
| shoulder +5 N m | −1.603 | +0.056 |
| shoulder −5 N m | −1.741 | −0.082 |
| elbow +5 N m | −1.679 | −0.020 |
| elbow −5 N m | −1.682 | −0.023 |
| zero torque | −1.744 | −0.085 |
$V(s_0) = -1.659 \pm 0.013$. The policy's own mean action has an advantage of zero within error, as it should: $V$ is the average of $Q$ over the policy's actions. A shoulder torque 5 N m higher has an advantage of +0.056, about three standard errors: at this state the learned policy is not greedy with respect to its own $Q$, and changing it toward that action would improve it. That is the idea every policy-gradient and actor-critic method builds on, and it needs only estimates of $A$, which is what PPO computes in Lesson 13.2.
## Implementation
```io
INPUT: arm2.xml
PROCESS: define the MDP; run the cross-entropy method on two policy classes with 5 seeds; estimate V and Q by Monte Carlo
OUTPUT: printed tables
```python file=examples/l13_1_policy_search.py “"”Lesson 13.1: an MDP in MuJoCo terms, a policy search you can watch, values and advantages.
INPUT arm2.xml (planar two-link arm, torque motors, a mocap target) PROCESS the task: bring the arm’s ee site to a random target in its plane within 2 s. Policy: torques u = clip(W_e e + W_v qdot + b), e = target - ee (x, z), held for 20 ms (10 physics steps); 10 parameters. With “gravity features” the policy also sees (cos q1, cos(q1 + q2)) through W_g: 14 parameters. Reward per control step: minus the distance (m); return = sum over 100 control steps. (1) the MDP spelled out: state, observation, action, reward, discount, horizon; (2) the cross-entropy method on the parameters (population 32, best 6 refitted, 4 shared random targets per iteration, 60 iterations), 5 seeds per policy class run in parallel: learning curves, the final mean policy on 100 held-out targets, and a Mann-Whitney test across seeds; (3) values: with Gaussian noise on the torques, Monte Carlo estimates of V(s0) and of Q(s0, a) for actions around the policy’s mean, and the advantages Q - V OUTPUT printed tables (the browser lab runs the same search)
Run: python examples/l13_1_policy_search.py “””
import math import os from multiprocessing import Pool
import mujoco import numpy as np from scipy.stats import mannwhitneyu
from mjcourse import model_path
FAST = os.environ.get(“MJC_FAST”) == “1” SKIP, STEPS = 10, 100 # physics steps per control step; control steps per episode (2 s) GAMMA = 0.99 SCALE = np.r_[np.full(4, 100.0), np.full(4, 10.0), np.full(2, 10.0), np.full(4, 10.0)] # N m/m, N m s/rad, N m, N m
def make(): model = mujoco.MjModel.from_xml_path(str(model_path(“arm2”))) return model, mujoco.MjData(model)
def sample_targets(rng: np.random.Generator, n: int) -> np.ndarray: “"”Targets in the arm’s plane: radius 0.35-0.85 m from the shoulder, angle -0.5 to 1.5 rad.””” r, a = rng.uniform(0.35, 0.85, n), rng.uniform(-0.5, 1.5, n) return np.c_[r * np.cos(a), 1.0 + r * np.sin(a)] # x, z (shoulder at z = 1)
def policy(theta: np.ndarray, e: np.ndarray, qdot: np.ndarray, q: np.ndarray) -> np.ndarray: “"”10 parameters: W_e, W_v, b. 14 parameters: also W_g on (cos q1, cos(q1 + q2)).””” p = theta * SCALE[:len(theta)] u = p[0:4].reshape(2, 2) @ e + p[4:8].reshape(2, 2) @ qdot + p[8:10] if len(theta) == 14: u = u + p[10:14].reshape(2, 2) @ np.array([math.cos(q[0]), math.cos(q[0] + q[1])]) return u
def rollout(model, data, theta, target, noise: float = 0.0, rng=None, first_action=None) -> tuple[float, float, list[float]]:
“"”Returns (return, final distance, rewards). first_action overrides the first control.”””
mujoco.mj_resetDataKeyframe(model, data, model.key(“bent”).id)
data.mocap_pos[0] = [target[0], 0.0, target[1]]
mujoco.mj_forward(model, data)
rewards = []
lo, hi = model.actuator_ctrlrange[:, 0], model.actuator_ctrlrange[:, 1]
for k in range(STEPS):
ee = data.site_xpos[0][[0, 2]]
u = policy(theta, target - ee, data.qvel, data.qpos) if first_action is None or k > 0 else first_action
if noise > 0 and (first_action is None or k > 0):
u = u + rng.normal(0.0, noise, 2)
data.ctrl[:] = np.clip(u, lo, hi)
for _ in range(SKIP):
mujoco.mj_step(model, data)
mujoco.mj_forward(model, data)
rewards.append(-float(np.linalg.norm(target - data.site_xpos[0][[0, 2]])))
return float(sum(rewards)), -rewards[-1], rewards
def cem(seed: int, iterations: int, size: int = 10, population: int = 32, elite: int = 6, targets_per: int = 4): rng = np.random.default_rng(seed) model, data = make() mean, std = np.zeros(size), np.full(size, 0.5) curve = [] for _ in range(iterations): targets = sample_targets(rng, targets_per) # shared by the whole population thetas = rng.normal(mean, std, (population, size)) returns = np.array([np.mean([rollout(model, data, th, t)[0] for t in targets]) for th in thetas]) best = thetas[np.argsort(returns)[-elite:]] mean, std = best.mean(axis=0), best.std(axis=0) + 0.01 curve.append((returns.mean(), np.sort(returns)[-elite:].mean(), returns.max())) return mean, curve
def evaluate(theta, targets) -> tuple[float, float, float]: “"”Success rate (final distance < 2 cm), median and 90th percentile of the final distance (m).””” model, data = make() finals = np.array([rollout(model, data, theta, t)[1] for t in targets]) return float(np.mean(finals < 0.02)), float(np.median(finals)), float(np.percentile(finals, 90))
def part1() -> None: model, _ = make() dt = model.opt.timestep * SKIP print(f” state q (2), qdot (2), target (2): what the simulator needs to continue”) print(f” observed e = target - ee (2), qdot (2): the policy does not see q, so gravity compensation can only be”) print(f” approximate (the observation is not a Markov state for this policy class)”) print(f” action shoulder and elbow torques, limits {model.actuator_ctrlrange[:, 1]} N m, held {1000 * dt:.0f} ms”) print(f” reward -|e| (m) per control step; episode {STEPS} control steps = {STEPS * dt:.1f} s”) print(f” discount gamma {GAMMA}: effective horizon 1/(1 - gamma) = {1 / (1 - GAMMA):.0f} steps = {dt / (1 - GAMMA):.1f} s”)
HELD_OUT = sample_targets(np.random.default_rng(1234), 100) # never used during the search
def search_and_evaluate(job: tuple[int, int, int]) -> tuple[int, int, np.ndarray, list, tuple]: size, seed, iterations = job theta, curve = cem(seed, iterations, size) return size, seed, theta, curve, evaluate(theta, HELD_OUT)
def part2() -> np.ndarray: iterations, seeds = (6, 1) if FAST else (60, 5) jobs = [(size, seed, iterations) for size in (10, 14) for seed in range(seeds)] if FAST: results = [search_and_evaluate(j) for j in jobs] else: with Pool(len(jobs)) as pool: results = pool.map(search_and_evaluate, jobs) marks = sorted({0, 9, 19, 39, iterations - 1} & set(range(iterations))) summary = {} for size, label in ((10, “observes e, qdot (10 parameters)”), (14, “also observes cos q1, cos(q1 + q2) (14 parameters)”)): print(f” policy {label}”) for _, seed, _, curve, (success, median, p90) in [r for r in results if r[0] == size]: print(f” seed {seed}: elite mean return at iterations “ + “, “.join(f”{i + 1}: {curve[i][1]:.1f}” for i in marks) + f”; held-out: success {success:.0%}, final distance median {1000 * median:.1f} mm, 90th percentile {1000 * p90:.1f} mm”) summary[size] = np.array([r[4] for r in results if r[0] == size]) if seeds > 1: for k, name in ((0, “success rate”), (1, “median final distance”)): test = mannwhitneyu(summary[14][:, k], summary[10][:, k], alternative=”two-sided”) print(f” {name} per seed, 14 against 10 parameters: medians {np.median(summary[14][:, k]):.3f} against “ f”{np.median(summary[10][:, k]):.3f}; Mann-Whitney U {test.statistic:.0f}, two-sided p = {test.pvalue:.3f}”) theta = next(r[2] for r in results if r[0] == 10 and r[1] == 0) print(f” parameters, first policy, seed 0 (W_e N m/m, W_v N m s/rad, b N m): {np.round(theta * SCALE[:10], 1)}”) return theta
def part3(theta: np.ndarray) -> None: model, data = make() target, noise, n = np.array([0.70, 1.13]), 1.0, (40 if FAST else 300) rng = np.random.default_rng(7) discounted = lambda rewards: sum(GAMMA**k * r for k, r in enumerate(rewards)) # noqa: E731 mujoco.mj_resetDataKeyframe(model, data, model.key(“bent”).id) data.mocap_pos[0] = [target[0], 0.0, target[1]] mujoco.mj_forward(model, data) mean_action = np.clip(policy(theta, target - data.site_xpos[0][[0, 2]], data.qvel, data.qpos), model.actuator_ctrlrange[:, 0], model.actuator_ctrlrange[:, 1]) start_distance = np.linalg.norm(target - data.site_xpos[0][[0, 2]]) v = [discounted(rollout(model, data, theta, target, noise, rng)[2]) for _ in range(n)] print(f” s0: the bent keyframe, target {target} ({100 * start_distance:.1f} cm away); “ f”torque noise sigma {noise} N m; {n} rollouts per estimate”) print(f” V(s0) = {np.mean(v):.3f} +- {np.std(v) / math.sqrt(n):.3f} (standard error)”) print(f” {‘first action a (N m)’:<26}{‘Q(s0, a)’:>10}{‘A = Q - V’:>11}”) for label, delta in ((“policy mean”, (0, 0)), (“mean + (5, 0)”, (5, 0)), (“mean - (5, 0)”, (-5, 0)), (“mean + (0, 5)”, (0, 5)), (“mean - (0, 5)”, (0, -5)), (“zero torque”, None)): a = np.zeros(2) if delta is None else mean_action + np.array(delta) q = [discounted(rollout(model, data, theta, target, noise, rng, first_action=a)[2]) for _ in range(n)] print(f” {label + ‘ ‘ + str(np.round(a, 1)):<26}{np.mean(q):>10.3f}{np.mean(q) - np.mean(v):>11.3f}”)
if name == “main”: print(f”MuJoCo {mujoco.version}”) print(“(1) the reach task as an MDP”) part1() print(“(2) cross-entropy method on a linear feedback policy”) theta = part2() print(“(3) value and action values of the learned policy, by Monte Carlo”) part3(theta)
Output:
```text
MuJoCo 3.14.0
(1) the reach task as an MDP
state q (2), qdot (2), target (2): what the simulator needs to continue
observed e = target - ee (2), qdot (2): the policy does not see q, so gravity compensation can only be
approximate (the observation is not a Markov state for this policy class)
action shoulder and elbow torques, limits [25. 15.] N m, held 20 ms
reward -|e| (m) per control step; episode 100 control steps = 2.0 s
discount gamma 0.99: effective horizon 1/(1 - gamma) = 100 steps = 2.0 s
(2) cross-entropy method on a linear feedback policy
policy observes e, qdot (10 parameters)
seed 0: elite mean return at iterations 1: -35.3, 10: -12.4, 20: -6.3, 40: -6.5, 60: -12.1; held-out: success 29%, final distance median 30.7 mm, 90th percentile 79.5 mm
seed 1: elite mean return at iterations 1: -46.9, 10: -11.8, 20: -7.3, 40: -5.7, 60: -4.8; held-out: success 59%, final distance median 18.1 mm, 90th percentile 28.1 mm
seed 2: elite mean return at iterations 1: -38.3, 10: -3.1, 20: -4.0, 40: -3.1, 60: -4.7; held-out: success 76%, final distance median 13.5 mm, 90th percentile 28.3 mm
seed 3: elite mean return at iterations 1: -33.1, 10: -5.9, 20: -3.1, 40: -4.8, 60: -4.8; held-out: success 87%, final distance median 11.6 mm, 90th percentile 20.9 mm
seed 4: elite mean return at iterations 1: -57.0, 10: -7.1, 20: -6.6, 40: -4.5, 60: -6.2; held-out: success 64%, final distance median 15.9 mm, 90th percentile 39.5 mm
policy also observes cos q1, cos(q1 + q2) (14 parameters)
seed 0: elite mean return at iterations 1: -34.2, 10: -10.9, 20: -19.6, 40: -4.0, 60: -2.7; held-out: success 83%, final distance median 6.9 mm, 90th percentile 56.0 mm
seed 1: elite mean return at iterations 1: -46.7, 10: -8.0, 20: -4.4, 40: -7.4, 60: -2.4; held-out: success 88%, final distance median 6.0 mm, 90th percentile 22.8 mm
seed 2: elite mean return at iterations 1: -33.3, 10: -9.8, 20: -6.3, 40: -6.8, 60: -5.6; held-out: success 45%, final distance median 20.8 mm, 90th percentile 81.3 mm
seed 3: elite mean return at iterations 1: -29.6, 10: -4.8, 20: -4.7, 40: -2.8, 60: -3.1; held-out: success 93%, final distance median 7.0 mm, 90th percentile 15.8 mm
seed 4: elite mean return at iterations 1: -56.9, 10: -9.2, 20: -5.0, 40: -2.7, 60: -2.6; held-out: success 85%, final distance median 7.9 mm, 90th percentile 24.3 mm
success rate per seed, 14 against 10 parameters: medians 0.850 against 0.640; Mann-Whitney U 19, two-sided p = 0.222
median final distance per seed, 14 against 10 parameters: medians 0.007 against 0.016; Mann-Whitney U 4, two-sided p = 0.095
parameters, first policy, seed 0 (W_e N m/m, W_v N m s/rad, b N m): [-101.6 109. 74.2 59.9 -7.2 -1.9 -3.6 -6.9 4.2 8.6]
(3) value and action values of the learned policy, by Monte Carlo
s0: the bent keyframe, target [0.7 1.13] (8.5 cm away); torque noise sigma 1.0 N m; 300 rollouts per estimate
V(s0) = -1.659 +- 0.013 (standard error)
first action a (N m) Q(s0, a) A = Q - V
policy mean [16.5 9.9] -1.661 -0.002
mean + (5, 0) [21.5 9.9] -1.603 0.056
mean - (5, 0) [11.5 9.9] -1.741 -0.082
mean + (0, 5) [16.5 14.9] -1.679 -0.020
mean - (0, 5) [16.5 4.9] -1.682 -0.023
zero torque [0. 0.] -1.744 -0.085
The search stalls at a poor return. The spread collapsed before the mean found good parameters: raise the population, raise the floor on $\sigma$, or start with a larger $\sigma$. Check the parameter scaling: one parameter whose useful range is 100 times another’s makes the Gaussian a poor search distribution.
The learning curve jumps up and down between iterations. Each iteration scores on new random targets, so some iterations are easier. Judge progress on a fixed held-out set.
A learned policy reaches every target with the same small offset. The policy cannot see what it would need to correct it (here, gravity). Change the observation or the policy class, not the optimizer.
Two runs of the same search give different conclusions. Seeds. Run several and report the spread.
Change the episode to 3 s and the success threshold to 1 cm, rerun part (2) with 5 seeds, and report whether the difference between the policy classes becomes significant. Explain why a longer episode should help the class that can compensate gravity more than the one that cannot.
Replace the cross-entropy method with a finite-difference policy gradient that uses the same rollouts: estimate the gradient of the mean return from antithetic pairs $\theta \pm \delta$ and take a step. Compare the number of rollouts each method needs to reach a held-out success of 80% with the 14-parameter policy, over 5 seeds.
[!research] Simple searches as baselines Methods that only see returns have been competitive with gradient-based reinforcement learning on MuJoCo locomotion benchmarks: evolution strategies (Salimans et al., arXiv:1703.03864) and random search over linear policies (Mania, Guy and Recht, arXiv:1803.07055). That says as much about the benchmarks as about the methods (inference). For a new task, a linear policy found by a cross-entropy or random search is a cheap baseline that shows whether a deep policy is needed at all, and whether the observation contains what the task requires: in this lesson, two features mattered more than any choice of optimizer.
{"id": "13.1-check", "title": "Knowledge check", "questions": [
{"kind": "numeric", "q": "With gamma = 0.98 and 20 ms control steps, what is the effective horizon in seconds?",
"answer": 1.0, "tol": 0.01, "unit": "s",
"explain": "<p>$1/(1-\\gamma) = 50$ steps, times 20 ms = 1 s.</p>"},
{"kind": "mcq", "q": "A policy observes the task-space error and joint velocities, not the joint angles. Why does it settle a few centimetres from targets?",
"options": ["The optimizer has not converged", "The gravity torque depends on the joint angles it cannot see, so its equilibrium balances gravity with an error", "The reward is too sparse", "The discount factor is too low"],
"answer": 1,
"explain": "<p>Its equilibrium satisfies $W_e e = g(q) - b$; with gravity features it can cancel $g(q)$ (output 2: median 7 mm against 16 mm).</p>"},
{"kind": "mcq", "q": "For a stochastic policy, the Monte Carlo estimate of Q at the policy's mean action is within one standard error of V. Is that expected?",
"options": ["No, Q should always exceed V", "Yes: V is the average of Q over the policy's actions, so its typical action has an advantage near zero", "No, V should exceed Q", "Only for deterministic policies"],
"answer": 1,
"explain": "<p>$V(s) = \\mathbb{E}_{a\\sim\\pi}[Q(s,a)]$; output (3) measured an advantage of −0.002 for the mean action.</p>"},
{"kind": "mcq", "q": "In output (3), raising the shoulder torque by 5 N m has an advantage of +0.056, about three standard errors. What does that mean for the policy?",
"options": ["The policy is optimal", "Changing the policy toward that action at this state would increase its return", "The value estimate is biased", "The noise is too large"],
"answer": 1,
"explain": "<p>A positive advantage is the direction of policy improvement; policy gradients follow it.</p>"}
]}
Lesson 13.2 writes PPO from scratch: advantages estimated by a learned value function, a clipped objective, and a seven-joint arm learning to reach.