Hands and bimanual cells are where models get large: a three-finger hand has six finger joints, two arms with grippers have sixteen actuators. Three tools keep them manageable.
Defaults let one definition serve many parts. hand3.xml defines a finger class once (joint damping, armature, range style, servo gains, capsule friction) and mounts three copies of the same finger with childclass="finger", differing only in where they are mounted.
Composition builds a scene from tested parts instead of copied text. The bimanual cell is two copies of arm7 with a gripper, attached by MjSpec into two frames with the prefixes left/ and right/, so every name stays unique: left/a1, right/gripper/grip.
Synergies give a high-dimensional hand a low-dimensional command. Instead of six joint targets, the lab’s hand takes one number $s$ from 0 (open) to 1 (closed), and a fixed map turns it into all six.
[!established] How MuJoCo applies defaults In MJCF,
<default class="...">sets attribute values for each element type; an element takes them from itsclassattribute, or from thechildclassof the nearest enclosing body; nested default classes inherit from their parent class. The values are applied when the model is parsed. InMjSpec, a default class is applied when an element is created: editing a class later changes elements created afterwards, not existing ones (output 1 shows both behaviours).
The synergy slider drives all six finger joints at once; Grasp runs the scripted routine from the script below: pre-shape the fingers open, descend, close along the synergy, lift.
```lab simlab {“dock”: true, “title”: “One number, six finger joints”, “model”: “hand3”, “height”: 260, “camera”: {“azimuth”: -60, “elevation”: 22, “distance”: 0.75, “target”: [0.05, 0.03, 0.1]}, “toggles”: [“contacts”, “forces”, “joints”], “overlays”: {“contacts”: true}, “forceScale”: 0.003, “setup”: “ctx.t0 = null”, “controls”: [ {“type”: “button”, “label”: “Grasp”, “run”: “mj.mj_resetData(model, data); const a = ctx.obj ? 16 : 9; data.qpos[a] += ctx.dx; ctx.t0 = data.time”}, {“type”: “select”, “label”: “object”, “param”: “obj”, “options”: [[“ball (radius 30 mm)”, 0], [“can (radius 25 mm)”, 1]], “value”: 0}, {“label”: “object offset in x”, “unit”: “m”, “param”: “dx”, “min”: -0.015, “max”: 0.015, “step”: 0.001, “value”: 0, “digits”: 3}, {“type”: “select”, “label”: “synergy”, “param”: “syn”, “options”: [[“whole finger (0.6, 1.0)”, 0], [“distal only (0.0, 1.6)”, 1], [“proximal only (1.0, 0.0)”, 2]], “value”: 0}, {“label”: “synergy s (manual)”, “param”: “s”, “min”: 0, “max”: 1, “step”: 0.01, “value”: 0, “digits”: 2}, {“label”: “palm height (manual)”, “unit”: “m”, “param”: “gz”, “min”: -0.2, “max”: 0.1, “step”: 0.005, “value”: 0, “digits”: 3}], “controller”: “const r = (x) => Math.min(Math.max(x, 0), 1); let s = ctx.s; const aim = ctx.obj ? [0.15, 0.1] : [0, 0]; data.ctrl[0] = aim[0]; data.ctrl[1] = aim[1]; data.ctrl[2] = ctx.gz; if (ctx.t0 != null) { const t = data.time - ctx.t0; data.ctrl[2] = t < 1.9 ? -0.18 * r((t - 0.6) / 1.0) : -0.18 + 0.15 * r((t - 3.4) / 1.0); s = r((t - 1.9) / 1.0); } const C = [[0.6, 1.0], [0.0, 1.6], [1.0, 0.0]][ctx.syn]; for (let f = 0; f < 3; f++) { data.ctrl[3 + 2 * f] = -0.3 + s * (C[0] + 0.3); data.ctrl[4 + 2 * f] = s * C[1]; }”, “readouts”: [ {“label”: “touch, finger 1 (N)”, “expr”: “data.sensordata[3]”, “digits”: 2}, {“label”: “touch, finger 2 (N)”, “expr”: “data.sensordata[4]”, “digits”: 2}, {“label”: “touch, finger 3 (N)”, “expr”: “data.sensordata[5]”, “digits”: 2}, {“label”: “object height (m)”, “expr”: “lib.bodyPos(ctx.obj ? ‘can’ : ‘ball’)[2]”, “digits”: 3}], “plots”: [{“ylabel”: “touch (N)”, “window”: 6, “traces”: [ {“label”: “finger 1 (N)”, “expr”: “data.sensordata[3]”}, {“label”: “finger 2 (N)”, “expr”: “data.sensordata[4]”, “dash”: true}, {“label”: “finger 3 (N)”, “expr”: “data.sensordata[5]”}]}], “note”: “Grasp resets the scene, moves the chosen object by the offset, and runs: open 0.6 s, descend 1 s, close along the synergy over 1 s, lift 15 cm. While it runs, the manual sliders are ignored; press Reset to return to manual control. A touch sensor sums the normal forces of contacts inside its fingertip site.”}
## Mathematics
### A synergy is a line through joint space
With pre-shape $q_\text{open}$ and closed pose $q_\text{closed}$, the hand's command is
$$q^\star(s) = q_\text{open} + s\,(q_\text{closed} - q_\text{open}), \qquad s \in [0, 1],$$
a segment in the six-dimensional space of finger joint targets. Every finger here gets the same (proximal, distal) pair, so the segment is three copies of a two-dimensional choice: the lesson compares three of them. With $k$ synergies the command becomes $q^\star = q_0 + S z$ with $S \in \mathbb R^{6 \times k}$, and the hand can reach a $k$-dimensional family of postures. Santello, Flanders and Soechting found that a few such components describe most of the variance of human grasp postures (*Postural Hand Synergies for Tool Use*, J. Neurosci. 1998, [doi:10.1523/JNEUROSCI.18-23-10105.1998](https://doi.org/10.1523/JNEUROSCI.18-23-10105.1998)), and Ciocarlie and Allen used such low-dimensional subspaces to plan robot grasps (*Hand Posture Subspaces for Dexterous Robotic Grasping*, IJRR 2009, [doi:10.1177/0278364909105606](https://doi.org/10.1177/0278364909105606)).
The price of the low dimension is visible in output (2): a synergy encodes one closing motion, and an object off-centre meets the fingers at different times. The distal-only synergy loses the ball for offsets of 9 mm or more towards $-x$, the gap between fingers 2 and 3, and keeps it for every offset towards finger 1. A plausible reading (inference; watch it in the lab): the distal links swing inward and upward about joints well above the ball's equator, so an off-centre ball is struck from above and pushed outward through the gap, while an offset towards finger 1 is pushed back into the hand by that finger.
### Symmetry and the touch forces
For a centred ball, the three fingers are images of one another under a 120° rotation, so their contact forces are equal: output (2) prints 19.17 N three times. When the readouts disagree with the ball centred, the fingers are not identical: a mounting error, a finger with different parameters, or, as this course found, a wrong quaternion (Debugging).
### Two arms, one object
When both grippers of the bimanual cell hold the long box, the box's six degrees of freedom are constrained by two grasps that can each transmit six components of force and torque. Six combinations of those twelve move the box; the other six squeeze, stretch or twist it without moving it. These **internal forces** are invisible in the box's motion and limited only by the grasps' friction, which is why bimanual controllers must regulate them explicitly (Project 19).
## Implementation
```io
INPUT: hand3.xml; arm7.xml and gripper.xml through mjcourse.model_builder
PROCESS: compare the three fingers' parameters and edit the shared class two ways; grasp the ball and the can with three synergies from 40 random offsets; compose the bimanual cell and attach a clashing prefix
OUTPUT: printed tables
```python file=examples/l5_4_hands_bimanual.py “"”Lesson 5.4: one finger reused three times, two arms composed into one scene.
INPUT hand3.xml (three-finger hand on a gantry), arm7.xml + gripper.xml (via mjcourse.model_builder) PROCESS (1) show that the three fingers share every parameter through the “finger” default class, and that editing the class edits all of them; (2) grasp the ball and the can with three one-dimensional synergies, from 40 randomized positions each; (3) compose the bimanual cell with MjSpec: names, counts, actuator lookup by name, collisions at home; then attach a second arm with a clashing prefix OUTPUT printed tables
Run: python examples/l5_4_hands_bimanual.py “””
import os
import mujoco import numpy as np
from mjcourse import model_builder, model_path, stats
N_TRIALS = 20 if os.environ.get(“MJC_FAST”) == “1” else 40 OPEN = np.array([-0.3, 0.0]) # pre-shape per finger: proximal spread out, distal straight SYNERGIES = { # closed pose per finger (proximal, distal), rad “whole finger (0.6, 1.0)”: np.array([0.6, 1.0]), “distal only (0.0, 1.6)”: np.array([0.0, 1.6]), “proximal only (1.0, 0.0)”: np.array([1.0, 0.0]), }
def finger_parameters() -> None:
model = mujoco.MjModel.from_xml_path(str(model_path(“hand3”)))
print(f” {‘joint’:<8}{‘damping’:>9}{‘armature’:>10}{‘range (rad)’:>16}{‘servo kp’:>10}{‘tip friction’:>14}”)
for f in (1, 2, 3):
for part in (“prox”, “dist”):
j = model.joint(f”f{f}{part}”)
a = model.actuator(f”f{f}{part}”)
tip = model.geom(f”f{f}tip”)
print(f” {j.name:<8}{j.damping[0]:>9.3f}{j.armature[0]:>10.3f}{np.array2string(j.range, precision=1):>16}”
f”{a.gainprm[0]:>10.1f}{tip.friction[0]:>14.2f}”)
xml = model_path(“hand3”).read_text()
line = ‘
spec = mujoco.MjSpec.from_file(str(model_path("hand3")))
finger = spec.find_default("finger")
finger.joint.damping[0] = 0.05 # edit the class after parsing...
probe = spec.worldbody.add_body(name="probe", pos=[0, 0, 1])
probe.add_joint(finger, name="probe") # ...then create one new joint from it
probe.add_geom(size=[0.01, 0, 0])
model = spec.compile()
damping = [float(model.joint(f"f{f}_{p}").damping[0]) for f in (1, 2, 3) for p in ("prox", "dist")]
print(f" edit the class through MjSpec: existing finger joints {damping}, "
f"a joint created afterwards {float(model.joint('probe').damping[0])}")
def grasp(closed: np.ndarray, obj: str, offset: np.ndarray) -> tuple[bool, np.ndarray]:
“"”Pre-shape, descend over the object’s nominal position, close along the synergy, lift 0.15 m.”””
model = mujoco.MjModel.from_xml_path(str(model_path(“hand3”)))
data = mujoco.MjData(model)
adr = model.jnt_qposadr[model.joint(obj).id]
nominal = data.qpos[adr:adr + 2].copy()
data.qpos[adr:adr + 2] = nominal + offset # the object is off-centre by offset
def run(seconds: float) -> None:
for _ in range(round(seconds / model.opt.timestep)):
mujoco.mj_step(model, data)
data.ctrl[0:2] = nominal # the hand aims at the nominal position
data.ctrl[3:9] = np.tile(OPEN, 3)
run(0.6)
for k in range(100):
data.ctrl[2] = -0.18 * (k + 1) / 100
run(0.01)
run(0.3)
for k in range(100): # one scalar s: 0 = pre-shape, 1 = closed
s = (k + 1) / 100
data.ctrl[3:9] = np.tile(OPEN + s * (closed - OPEN), 3)
run(0.01)
run(0.5)
touch = data.sensordata[3:6].copy()
z0 = data.xpos[model.body(obj).id][2]
for k in range(100):
data.ctrl[2] = -0.18 + 0.15 * (k + 1) / 100
run(0.01)
run(1.0)
return bool(data.xpos[model.body(obj).id][2] - z0 > 0.12), touch
def synergies() -> None: rng = np.random.default_rng(5) offsets = rng.uniform(-0.015, 0.015, size=(N_TRIALS, 2)) ok, touch = grasp(SYNERGIES[“whole finger (0.6, 1.0)”], “ball”, np.zeros(2)) print(f” centred ball, whole-finger synergy: lifted {ok}, fingertip touch forces {np.round(touch, 2)} N”) print(f” {N_TRIALS} positions with the object off-centre by up to 15 mm in x and y:”) for name, closed in SYNERGIES.items(): cells = [] for obj in (“ball”, “can”): wins = sum(grasp(closed, obj, off)[0] for off in offsets) lo, hi = stats.wilson_interval(wins, N_TRIALS) cells.append(f”{obj} {wins:2d}/{N_TRIALS} [{lo:.2f}, {hi:.2f}]”) print(f” {name:<26} {‘ ‘.join(cells)}”) print(“ (success: lifted by more than 12 cm with the palm’s 15 cm lift; 95% Wilson intervals)”) xs = (-0.015, -0.012, -0.009, -0.006, 0.006, 0.009, 0.012, 0.015) print(f” ball offset along x only (finger 1 sits at +x), offsets {[round(1000 * x) for x in xs]} mm:”) for name, closed in SYNERGIES.items(): row = ““.join(“+” if grasp(closed, “ball”, np.array([x, 0.0]))[0] else “.” for x in xs) print(f” {name:<26} {row} (+ lifted, . lost)”)
def bimanual() -> None: spec = model_builder.build_bimanual() model = spec.compile() data = mujoco.MjData(model) print(f” bodies {model.nbody}, joints {model.njnt}, actuators {model.nu}, nq {model.nq}, nv {model.nv}”) print(f” actuators: {[model.actuator(i).name for i in range(model.nu)]}”) left = model.actuator(“left/gripper/grip”).id right = model.actuator(“right/gripper/grip”).id print(f” the two gripper commands by name: ctrl[{left}] and ctrl[{right}]”) mujoco.mj_resetDataKeyframe(model, data, model.key(“home”).id) mujoco.mj_forward(model, data) pairs = sorted({“ / “.join(sorted((model.geom(c.geom1).name, model.geom(c.geom2).name))) for c in data.contact[:data.ncon]}) print(f” contacts at the home keyframe: {pairs}”)
scene = model_builder.build_bimanual() # attach a third arm with a prefix already in use
extra = model_builder.build_arm7_gripper()
frame = scene.worldbody.add_frame(name="extra_mount", pos=[1.2, 0, 0])
try:
frame.attach_body(extra.body("link0"), "left/", "")
scene.compile()
print(" a second 'left/' arm compiled (unexpected)")
except Exception as err: # MuJoCo reports the clash; record its message
print(f" attaching another arm with prefix 'left/': {type(err).__name__}: {str(err).splitlines()[0]}")
if name == “main”: print(“(1) one finger definition, three fingers:”) finger_parameters() print(“(2) one-dimensional synergies:”) synergies() print(“(3) a bimanual cell from two copies of one arm:”) bimanual()
Output:
```text
(1) one finger definition, three fingers:
joint damping armature range (rad) servo kp tip friction
f1_prox 0.020 0.001 [-0.3 1.4] 2.0 1.50
f1_dist 0.020 0.001 [0. 1.6] 2.0 1.50
f2_prox 0.020 0.001 [-0.3 1.4] 2.0 1.50
f2_dist 0.020 0.001 [0. 1.6] 2.0 1.50
f3_prox 0.020 0.001 [-0.3 1.4] 2.0 1.50
f3_dist 0.020 0.001 [0. 1.6] 2.0 1.50
edit the class in the XML, recompile: finger joint damping [0.05, 0.05, 0.05, 0.05, 0.05, 0.05]
edit the class through MjSpec: existing finger joints [0.02, 0.02, 0.02, 0.02, 0.02, 0.02], a joint created afterwards 0.05
(2) one-dimensional synergies:
centred ball, whole-finger synergy: lifted True, fingertip touch forces [19.17 19.17 19.17] N
40 positions with the object off-centre by up to 15 mm in x and y:
whole finger (0.6, 1.0) ball 39/40 [0.87, 1.00] can 40/40 [0.91, 1.00]
distal only (0.0, 1.6) ball 15/40 [0.24, 0.53] can 40/40 [0.91, 1.00]
proximal only (1.0, 0.0) ball 40/40 [0.91, 1.00] can 40/40 [0.91, 1.00]
(success: lifted by more than 12 cm with the palm's 15 cm lift; 95% Wilson intervals)
ball offset along x only (finger 1 sits at +x), offsets [-15, -12, -9, -6, 6, 9, 12, 15] mm:
whole finger (0.6, 1.0) ++++++++ (+ lifted, . lost)
distal only (0.0, 1.6) ...+++++ (+ lifted, . lost)
proximal only (1.0, 0.0) ++++++++ (+ lifted, . lost)
(3) a bimanual cell from two copies of one arm:
bodies 24, joints 19, actuators 16, nq 25, nv 24
actuators: ['left/a1', 'left/a2', 'left/a3', 'left/a4', 'left/a5', 'left/a6', 'left/a7', 'left/gripper/grip', 'right/a1', 'right/a2', 'right/a3', 'right/a4', 'right/a5', 'right/a6', 'right/a7', 'right/gripper/grip']
the two gripper commands by name: ctrl[7] and ctrl[15]
contacts at the home keyframe: ['box / table']
attaching another arm with prefix 'left/': ValueError: Error: repeated name 'left/gripper/grip' in tendon
Reading the output:
MjSpec after parsing changes none of them, only a joint created afterwards: in MjSpec, a default is a template applied at creation, not a live link. To change existing elements, change the elements, or edit the XML and re-parse.model.actuator("right/gripper/grip").id), never by position: the right gripper’s index, 15, is an accident of attachment order. Attaching another arm under a prefix already in use fails at compile time with the name of the first clash.Two fingers of a hand curl sideways into each other. Their mounting frames are yawed by the wrong angles, so each finger’s flexion axis is tangential instead of radial. Print each finger’s local $x$ axis against its mounting direction from the palm centre. The course’s own hand3.xml had exactly this bug: the quaternions of fingers 2 and 3 were swapped (240° and 120° instead of 120° and 240°), so they closed into each other and even touched at rest. Nothing in compilation or in Lesson 2.2’s use of the model revealed it; the grasp experiment in this lesson did. The file is fixed and now states the rule: for a yaw $a$, quat = "cos(a/2) 0 0 sin(a/2)".
repeated name ... in tendon when composing models. Two attached copies share a prefix (output 3). Give every copy its own.
An edit to a default class through MjSpec has no effect. Defaults apply when elements are created (output 1). Edit the elements, or edit the MJCF and re-parse.
A controller written for one arm drives the wrong joints in the bimanual scene. It used actuator indices. Use names, or compute the index map once from names at start-up and test it.
The hand pins the object to the floor during the approach. With straight fingers, the fingertips sit inside the ball’s radius, so the descending hand lands on the ball. Pre-shape the fingers open before the approach, as the script does with OPEN.
Add a second synergy to the lab’s controller: $q^\star = q_0 + s_1 S_1 + s_2 S_2$ with $S_1$ the whole-finger closing and $S_2$ a pure distal curl. Find, by hand in the lab, a pair $(s_1, s_2)$ that lifts the ball from a 15 mm offset in $x$ with all three touch readouts above 5 N.
Learn synergies from data. For ball radii from 20 to 40 mm and random offsets, search for a closed pose (six joint targets, fingers not forced to be equal) that lifts the ball with the most balanced touch forces. Collect 200 successful poses, run principal component analysis on them, and report how many components explain 90% of the variance. Then grasp new balls using only those components and compare the success rate with the one-dimensional synergies of this lesson.
[!research] How many dimensions does a hand need to be commanded in? Dexterous-manipulation policies face the choice made in this lesson: act in the full joint space, or in a learned or designed low-dimensional subspace. The low-dimensional route is easier to explore and to teleoperate and, as output (2) shows, it can be very reliable within the conditions it was designed for and fail outside them. The bimanual cell raises the analogous question for two arms, with the added burden of internal forces. Low-cost bimanual data collection, such as the system of Zhao et al. on the Reading list, composes two arms exactly as this lesson does and leaves the coordination to the learned policy.
{"id": "5.4-check", "title": "Knowledge check", "questions": [
{"kind": "mcq", "q": "You load a hand with MjSpec, set <code>spec.find_default('finger').joint.damping[0] = 0.05</code>, and compile. What damping do the existing finger joints have?",
"options": ["0.05: the class is applied at compile time", "Their old value: MjSpec applies a default when an element is created", "Zero: editing a class clears it", "Compilation fails"],
"answer": 1,
"explain": "<p>Output (1): the six finger joints kept 0.02 while a joint created after the edit received 0.05. Editing the class in the MJCF and re-parsing changes all of them.</p>"},
{"kind": "mcq", "q": "Three identical fingers are mounted at 0°, 120° and 240° around a palm. Which quaternion mounts the finger at 120°, so that its local x axis points radially outward?",
"options": ["1 0 0 0", "0.5 0 0 0.866", "-0.5 0 0 0.866", "0.866 0 0 0.5"],
"answer": 1,
"explain": "<p>A yaw $a$ about $z$ is $(\\cos\\tfrac a2, 0, 0, \\sin\\tfrac a2)$; for $a$ = 120° that is (0.5, 0, 0, 0.866). The third option is 240°, the bug this lesson found in <code>hand3.xml</code>.</p>"},
{"kind": "numeric", "q": "Two grippers rigidly hold one box. Each grasp can transmit a full wrench (6 components). How many independent internal force components can the grippers apply without moving the box?",
"answer": 6, "tol": 0, "unit": "",
"explain": "<p>Twelve wrench components in total, six of which are needed to move or support the box's six degrees of freedom; the remaining six squeeze, stretch, shear or twist it.</p>"},
{"kind": "mcq", "q": "With 40 trials each, synergy A lifted the ball 39 times and synergy B 40 times. What does this show?",
"options": ["B is better", "A is better, because it touched the ball more gently", "Nothing about which is better: their 95% intervals, [0.87, 1.00] and [0.91, 1.00], overlap almost entirely", "Both succeed 100% of the time"],
"answer": 2,
"explain": "<p>One failure in 40 is indistinguishable from none at this sample size. Research mode shows how many trials a given difference needs.</p>"}
]}
Level 6 computes where the end effector is, how fast it moves, and how to get it where you want: forward kinematics, Jacobians, and inverse kinematics on the arms built in this level.