When an autonomous policy fails an episode, recording a binary zero does not explain what went wrong. Did the vision encoder fail to detect the target object? Did language attention bind to the wrong entity? Did spatial planning misjudge clearances? Did the action head produce erratic commands? Or did physical contact slip at the surface?
In physical deployments, answering these questions requires laborious manual video inspection or privileged instrumentation. In MuJoCo, we can answer them with causal rigor through counterfactual intervention.
We replay each failed episode from its exact initial state, substituting ground-truth oracle signals for one pipeline stage at a time:
\[\mathrm{do}(\text{stage} = \text{oracle})\]The earliest stage whose repair converts the failure into a success is identified as the causal failure mode. This attribution separates perceptual errors from language grounding and physical control.
The attribution procedure traverses a hierarchical tree of counterfactual interventions, ordered from low-level execution to high-level semantic reasoning:
Let an episode rollout be parameterized by initial simulator state $s_0$, visual observation process $O$, language prompt $\ell$, spatial coordinate representation $Z$, policy action generation $\pi$, and physics transition dynamics $T$. The binary episode outcome is $Y \in {0, 1}$.
For a failed episode where $Y(s_0, O, \ell) = 0$, we apply a sequence of interventions:
\[\tau_k = \mathrm{do}(X_k = X_k^*)\]where $X_k^*$ represents the privileged oracle implementation of stage $k$. The causal attribution index is defined as:
\[k^* = \arg\min_k \left\{ k \in \{1, \dots, K\} : Y(\tau_k) = 1 \right\}\]Ordering matters. We test lower-level mechanical and perceptual sufficiency before attributing failure to high-level cognitive ambiguity. If a policy fails simply because its gripper slipped on an edge, diagnosing it as a language comprehension failure would be incorrect.
INPUT: runs/vla/eval_patch.json (evaluation rollouts), trained MiniVLA weights, oracle MiniVLA
PROCESS: replay each failure under counterfactual stage repairs; record successful interventions
OUTPUT: runs/vla/diagnosis_patch.json and stage attribution breakdown table
```python file=examples/l22_6_failure_analysis.py import argparse import json import multiprocessing import os from collections import Counter from concurrent.futures import ProcessPoolExecutor from pathlib import Path
import torch
from mjcourse.vla import diagnosis from mjcourse.vla import evaluation as ev
RUNS = Path(file).resolve().parents[1] / “runs” / “vla” _MODELS: dict = {}
def _load(path: str): if path not in _MODELS: _MODELS[path] = torch.load(path, weights_only=False) return _MODELS[path]
def job(args) -> dict: vision, record = args torch.set_num_threads(1) seed = record[“model_seed”] model = load(str(RUNS / f”minivla{vision}_s{seed}_pos.pt”)) oracle = _load(str(RUNS / f”minivla_oracle_s{seed}_pos.pt”)) return {**diagnosis.diagnose(record, model, oracle), “condition”: record[“condition”], “seed”: seed, “text”: record[“text”]}
FAST = os.environ.get(“MJC_FAST”) == “1”
def main() -> None: parser = argparse.ArgumentParser() parser.add_argument(“–vision”, default=”patch”, choices=[“patch”, “keypoint”]) args = parser.parse_args() records = json.loads((RUNS / f”eval_{args.vision}.json”).read_text()) failures = [r for r in records if not r[“success”]] if FAST: failures = failures[:4] workers = int(os.environ.get(“SLURM_CPUS_PER_TASK”, min(32, os.cpu_count() or 1))) with ProcessPoolExecutor(workers, mp_context=multiprocessing.get_context(“spawn”)) as pool: verdicts = list(pool.map(job, [(args.vision, r) for r in failures], chunksize=2)) (RUNS / f”diagnosis_{args.vision}.json”).write_text(json.dumps(verdicts)) print(f”{len(failures)} failures of {len(records)} episodes (MiniVLA, {args.vision}), attributed to the first repaired stage”) print(f” {‘condition’:<16}{‘failures’:>9}” + ““.join(f”{s:>20}” for s in diagnosis.STAGES)) for c in ev.CONDITIONS: counts = Counter(v[“stage”] for v in verdicts if v[“condition”] == c) n = sum(counts.values()) print(f” {c:<16}{n:>9}” + ““.join(f”{counts[s]:>20}” for s in diagnosis.STAGES)) total = Counter(v[“stage”] for v in verdicts) print(f” {‘all’:<16}{len(verdicts):>9}” + ““.join(f”{total[s]:>20}” for s in diagnosis.STAGES)) print(f”wrote {RUNS / f’diagnosis_{args.vision}.json’}”)
if name == “main”: main()
## Measured results
Running counterfactual diagnosis across all failed evaluation episodes from Lesson 22.5 reveals distinct failure signatures across conditions:
| Condition | Total Failures | Control | Perception | Language Grounding | Spatial Reasoning | Action Drift | Contact Slip |
|---|---|---|---|---|---|---|---|
| in distribution | 15 | 0 | 2 | 0 | 1 | 9 | 3 |
| paraphrase | 54 | 0 | 4 | 29 | 3 | 12 | 6 |
| synonym | 243 | 0 | 6 | 218 | 2 | 11 | 6 |
| composition | 54 | 0 | 3 | 8 | 19 | 18 | 6 |
| larger pucks | 27 | 2 | 1 | 0 | 2 | 7 | 15 |
| recoloured | 135 | 0 | 108 | 0 | 8 | 14 | 5 |
| near the rim | 111 | 3 | 24 | 0 | 21 | 48 | 15 |
| crowded | 69 | 1 | 7 | 0 | 44 | 12 | 5 |
| camera shifted | 51 | 0 | 38 | 0 | 2 | 8 | 3 |
| slower pusher | 99 | 74 | 2 | 0 | 1 | 18 | 4 |
### Diagnostic insights
Language shifts are dominated by semantic grounding. In the synonym condition, 218 of 243 failures (90%) are cured immediately when replacing the synonym with the canonical prompt. The visual tokens and action head were entirely capable of executing the task; the failure lay strictly in the word embedding lookup.
Photometric shifts isolate perception. In the recoloured puck condition, 80% of failures are resolved by providing oracle object positions. The language grounding was correct, but visual attention tokens drifted between the target puck and distractors due to color similarity.
Cluttered scenes trigger spatial confusion. In the crowded condition, where pucks start within 6.5 to 10 cm of one another, 64% of failures are resolved by spatial trajectory intervention. The policy successfully navigates toward the cluster but collides with adjacent objects before reaching the target.
Embodiment mismatches expose control timeouts. In the slower pusher condition, 75% of failures are resolved by removing the actuator velocity throttle. The policy trajectory was geometrically sound, but execution speed fell below the required threshold for episode completion.
### Failure analysis on the 7-DOF Franka manipulator
On the 7-DOF Franka manipulator (`franka_vla.xml`), counterfactual replay isolates manipulation bottlenecks that do not exist on planar pushers:
- Grasp alignment slip: The gripper fingers contact the cube corners rather than opposite parallel faces, causing the cube to pivot out of the grip during lifting.
- Nullspace drift: Successive Cartesian delta commands pull the elbow joint toward its joint limit, causing singularity deceleration.
- Dual-view occlusion recovery: When an overhead view is blocked by arm self-occlusion during a descent, the wrist camera provides the necessary counterfactual recovery signal.
## Debugging
**Masking by broad interventions.** If an early stage in the diagnostic tree is too permissive (for instance, replacing actions with an optimal global planner too early), it will mask downstream failures. Ensure that each intervention repairs only its dedicated sub-problem.
**Non-deterministic replay drift.** Replaying a simulation episode requires setting the exact floating-point initial state: generalized coordinates, velocities, and numerical integrator state (`mjtState.mjSTATE_INTEGRATION`). If contact points are not preserved, chaotic divergences will invalidate the diagnosis.
## Exercise
Select ten failed episodes from the "near the rim" condition. Inspect the terminal puck coordinates and compute the fraction of failures caused by wall contact versus pusher reaching limits.
## Challenge
Design a counterfactual intervention that repairs depth perception by projecting 3D point clouds into the policy observation space. Test whether depth grounding eliminates visual perception failures in the camera shifted condition.
## Research connection
> [!research] Counterfactual diagnosis in robotic foundation models
> Analyzing failure modes on physical hardware is bottlenecked by the inability to reset identical physical conditions. Benchmarks such as SimplerEnv (Li et al., arXiv:2405.05941) attempt to replicate real-world Franka and WidowX evaluation in simulation. Implementing automated counterfactual attribution pipelines enables researchers to quantify whether policy degradation in new domains stems from visual domain gap, prompt misunderstanding, or physical impedance mismatch.
```quiz
{"id": "22.6-check", "title": "Knowledge check", "questions": [
{"kind": "mcq", "q": "What is the defining mechanism of counterfactual failure analysis in simulation?",
"options": ["Running longer training runs", "Replaying a failed episode while substituting ground truth for exactly one stage at a time", "Averaging losses across all seeds", "Adding noise to observations"],
"answer": 1,
"explain": "<p>Counterfactual analysis intervenes on one pipeline component while holding all other states and inputs identical to isolate causal attribution.</p>"},
{"kind": "mcq", "q": "In the synonym condition, which stage accounted for over 85% of failures?",
"options": ["Actuator control", "Language grounding", "Contact slip", "Physics integration"],
"answer": 1,
"explain": "<p>Synonyms introduced out-of-vocabulary tokens that failed semantic grounding; substituting canonical prompts cured the failures immediately.</p>"},
{"kind": "mcq", "q": "Why must low-level controller checks precede high-level semantic checks in the diagnostic tree?",
"options": ["To avoid falsely attributing physical tracking failures to cognitive misunderstanding", "Controllers take longer to simulate", "Perception is always correct", "Torque commands never saturate"],
"answer": 0,
"explain": "<p>If an actuator is saturated or lagging, high-level planning cannot succeed; ruling out mechanical insufficiency ensures semantic attribution is valid.</p>"},
{"kind": "mcq", "q": "On a 7-DOF Franka manipulator, what failure mode is distinct from planar pusher failures?",
"options": ["Grasp alignment slip and kinematic singularities", "Overhead camera rendering", "Seed initialization", "Token batching"],
"answer": 0,
"explain": "<p>Multi-joint arms involve 3D grasping stability, parallel finger friction cones, and kinematic Jacobian singularities that do not exist in planar pushing.</p>"}
]}
Lesson 22.7 synthesizes the architectural, evaluative, and diagnostic foundations of Level 22 into a complete, pre-registered VLA research project.