Where the human-video to robot-skill pipeline actually breaks
NVIDIA's Video-to-Data (V2D) Challenge splits the human-video to robot-policy pipeline into three coupled tracks and asks which stage loses the task-relevant signal: 4D reconstruction from monocular third-person video, robotic grounding of that reconstruction into a policy, and egocentric video to policy end to end.
Why it matters
The field reports reconstruction quality and policy success in separate papers with separate protocols, so nobody has shown that lowering chamfer distance raises manipulation success. V2D fixes the embodiment, the task suite and the corruption model across all three tracks, which turns that question into a measurable one for the first time.
My angle
The tier structure of Track 2 is the interesting instrument, not the leaderboard. Tier 1 uses clean multi-view captures, Tier 2 injects corruption sampled from the Track 1 error distribution, and Tier 3 uses off-the-shelf reconstructions. Sweeping that corruption gives a dose-response curve from perceptual error to policy success, which is the same question the outcome-bottleneck work asks: which part of the perceptual error is task-relevant at all. The cost is real. Track 2 fixes a G1 humanoid with Dex3 hands, so entering it means standing up an Isaac Lab humanoid retargeting and RL stack that has nothing in common with the Franka and LIBERO tooling already running here, and the leaderboard freezes on 4 Nov 2026, about six weeks out, inside the CVPR 2027 window. Track 3 is the cheaper fit: it is method-agnostic and accepts an implicit egocentric-video-to-policy model, which is the world-model direction already funded at Future Drive.
From the source
- Track 1
- 4D reconstruction of human-object interaction from a single third-person RGB video, in metric scale, no depth sensor or rigging. Scored equally on accuracy (chamfer distance for human and object meshes against a multi-view baseline) and physical plausibility (acceleration error, contact penetration).
- Track 2
- Robotic grounding of 4D human-object interaction into a policy on a fixed G1 robot with Dex3 hands. Three tiers: clean multi-view capture, synthetic corruption (jitter, dropout, contact error) sampled from Track 1 error distributions, and off-the-shelf reconstructions. Reported with AUC, SP-SR, MP-SR and MPPE from the CHORD manuscript.
- Track 3
- Egocentric video to policy, evaluated end to end and method-agnostic: explicit reconstruct-then-retarget or implicit VLA / world-action model. Held-out test set curated from an in-house collection.
- Timeline
- Leaderboard opens with the data release, freezes 4 Nov 2026 at 17:00 EST, winners announced 12 Nov 2026 at the CoRL 2026 tutorial. Up to five submissions per week, unlimited in the final three days.
- Eligibility
- A valid final submission, code released under a research license, and a technical report contribution. Nine awards: winner, runner-up and an industry innovation award per track.
Open questions
- Does Track 1 accuracy rank-correlate with Track 2 and Track 3 policy success? Kendall tau across public submissions answers it, and a weak correlation would mean the reconstruction leaderboard optimises a metric the downstream task does not care about.
- Which corruption mode dominates the drop from Tier 1 to Tier 3: pose jitter, temporal dropout, or contact error? The Tier 2 sampler is parameterised, so this is an ablation rather than a guess.
- Does training the grounding stage on Tier 2 corruptions beat training on clean Tier 1 data when both are evaluated on Tier 3?
- Is metric-scale recovery the binding constraint, or is contact and wrench plausibility the thing that decides whether a retargeted motion survives on hardware?
- Does any conclusion drawn on a G1 with Dex3 hands transfer to a five-finger gripper on a mobile base, which is the RISE deployment target?
Next actions
- Register and pull the released data and evaluation scripts.
- Read the Tier 2 corruption sampler in eval_robotic_grounding.py and record exactly which error statistics it draws from Track 1.
- Check whether the leaderboard exposes per-episode results. Without them the correlation study needs its own submissions.
- Decide scope before committing compute: Track 3 only, or no entry and an analysis paper built on the released error distributions.
Touches my own work
Facts above were read from the challenge page on 21 Sep 2026. The page prints "September 21, 2025" for the data release, which is a typo for 2026 given the CoRL 2026 cycle. Confirm every date on the challenge site before committing to it.