V-JEPA, V-JEPA 2-AC and H-JEPA, taken apart

Twenty-one premises that circulate about these models, including four that earlier versions of this page got wrong; items 15 to 21 are new on 12 Oct 2026. Each is checked against the paper, the code at commit 204698b, or a named primary source. Tags: Paper the V-JEPA 2 paper, Code the official repository or numbers measured by running it, H-JEPA the H-JEPA paper and its code, Context another primary source, Inference and Speculation my own reasoning.

The energy view

LeCun's 2022 position paper recasts prediction as an energy-based model: a scalar compatibility E(x, y) that is low when the continuation y fits the context x and high otherwise, with no requirement that it normalise into a density. A joint-embedding predictive architecture computes that energy in representation space. An encoder maps both sides, a predictor carries the context forward under a conditioning variable z, and a distance scores the match. LeCun 2022, §4

The argument against pixel prediction is a decomposition. For any predictor ŷ of y given x, E‖y − ŷ‖² = ‖E[y|x] − ŷ‖² + tr Cov(y|x). The second term is irreducible, and in natural video it is dominated by content nothing can forecast: foliage, water, sensor noise, specular highlights. A likelihood objective still hands out gradient in proportion to per-pixel error, so capacity drifts toward the nuisance. A JEPA pays only for what survives the encoder, and the encoder is free to drop what it cannot predict. The paper states the same intuition with "each blade of grass in a field". Paper §1 Littwin et al. (NeurIPS 2024) prove a matching statement for deep linear networks: latent prediction prioritises features with large regression coefficients, reconstruction prioritises high-variance ones. Littwin et al. 2024

Step 1 of the tour turns the decomposition into a slider. open step 1 →

The bet, and what would falsify it

The stated target is to "learn to understand the world and learn to act largely by observation". The recipe splits the learning in two: action-free pretraining on over a million hours of internet video, then action-conditioned post-training on "under 62 hours" of one robot's logs, a figure most likely counted by the video files' clock, which runs four times fast (section 05). Paper abstract, Fig. 1

The split is the central bet because interaction data is the scarce resource. If the representation learned from watching carries the state a controller needs, a small amount of interaction is enough to attach actions to it. Three results would falsify it. First, an action-conditioned predictor trained on the same 62 hours over an image encoder (DINOv2, as in DINO-WM) or a random one matching V-JEPA 2-AC's success would mean the video pretraining is not what carries the result; the paper runs no such encoder ablation for the AC stage. Second, success that scales with robot hours but not with pretraining scale. Third, a behaviour-cloning policy on the same 62 hours matching it. Octo, trained on all of DROID with hindsight goals, loses, which supports the bet but changes architecture, objective and pretraining at once. Inference

Every model on this page makes the same move: encode what is visible, predict the encoding of what is hidden or still to come, and score the prediction in a space the encoder learned. What changes from one model to the next is five design choices: what is hidden (image blocks, video tubes, the next frame, a state several steps ahead), what the prediction is conditioned on (token positions, actions, pooled macro-actions), the distance (L2 or L1), how collapse is prevented (a slow teacher network, or a regulariser on the batch, or an action-regression head), and what the result is used for (frozen features, or a simulator a planner searches). Click a model on the map for its recipe, its headline number and the protocol behind that number. Inference

One notation for the whole family

Write E_θ for the online encoder, E_θ̄ for the target encoder, P_φ for the predictor, sg for stop-gradient and M for the hidden tokens. The objectives then read:

Which "H-JEPA"?

Three different things carry the name, and citing the wrong one is easy. This page means the first item of the third group whenever it writes H-JEPA without qualification.

Position enters through a 3D rotary embedding: each head's feature dimension is split into three near-equal segments, rotated by the temporal, height and width index respectively. The paper reports it stabilised training of the largest models, relative to V-JEPA's absolute sin-cos table. Paper §2.1 In code each axis gets 2·⌊⌊d_head/3⌋/2⌋ dimensions, so a 64-dim ViT-g head rotates 20 + 20 + 20 and leaves 4 untouched, and the 32-dim pretraining predictor head rotates 10 + 10 + 10 and leaves 2. src/models/utils/modules.py, RoPEAttention

One pretraining step, with shapes

ViT-g, a 16-frame 256 px clip, the short-range mask, B clips per GPU. |C| and |M| are the context and target counts after the collator's batch-minimum truncation, measured at B = 24.

The objective and the teacher

Eq. 1 of the paper. Δ_y is a learnable mask token placed at each dropped position, the loss is taken only over masked positions, and the teacher weights θ̄ are an exponential moving average of the student's. A fixed m of 0.99925 averages over about 1/(1 − m) ≈ 1,333 steps. Paper Eq. 1, Table 9

Why it does not collapse

The prediction term alone is minimised by a constant encoder, so every JEPA needs a mechanism that keeps the representation from going flat. V-JEPA 2 relies on three: a stop-gradient on the target branch, so the target is never pulled toward the prediction; an EMA teacher, a lagged and non-degenerate copy; and an asymmetric predictor, which gives the online branch a degree of freedom the target lacks. Tian, Chen and Ganguli (ICML 2021) show for linear networks that the predictor comes to share eigenvectors with the representation's correlation matrix and that eigenmodes above a threshold grow rather than decay. SimSiam (Chen and He, CVPR 2021) shows stop-gradient plus a predictor suffice without an EMA. None of this is a guarantee: partial dimensional collapse is common in non-contrastive methods. Tian et al. 2021; Chen and He 2021

What V-JEPA 2 shows about collapse empirically: nothing. The paper asserts that the stop-gradient and EMA "prevent representation collapse" and reports no rank, spectrum or variance diagnostic. Paper §2.1

The contrast is with methods that regularise the batch directly. VICReg hinges each dimension's standard deviation above a floor and penalises off-diagonal covariance. SIGReg, used by LeWM, constrains the batch to an isotropic Gaussian by testing 1-D projections against the normal characteristic function (M = 1024 projections, weight 0.1). It removes the EMA, the stop-gradient and the pretrained encoder and trains a 15M-parameter world model from pixels, at a price: every dimension must carry unit variance. On the 2-D Two-Room task LeWM reaches 87% where the other planners reach 97% to 100%, which its authors attribute to SIGReg in very low-complexity environments. LeWM arXiv 2603.19312, §3 and Fig. 6

Step 5 runs a toy linear JEPA live and lets you remove each mechanism. open step 5 →

Masking, measured

Both masks are unions of rectangular blocks that span the full clip in time, so each is a set of tubes. Block size is drawn once per batch (spatial scale 0.15 or 0.7 of the frame, aspect ratio in [0.75, 1.5]) and block positions per clip. The collator then truncates every clip's context and target index lists to the batch minimum. The truncation keeps the first indices in (t, h, w) order, so a heavily masked clip loses its last time slices from the target, and a lightly masked clip loses its last visible time slices from the context. src/masks/multiseq_multiblock3d.py

Why tubes: adjacent frames are nearly redundant, so a masked token with a visible copy at t ± 1 can be interpolated rather than inferred. V-JEPA's ablation found multiblock beats random-tube masking at 90%, which produced "features of low semantic quality", and beats causal masking from early frames only. V-JEPA §4 Because the context contains frames on both sides of every masked step, this is spatiotemporal inpainting, not extrapolation: V-JEPA 2 pretraining produces a state encoder, and the explicit predict-the-next-frame objective appears only in V-JEPA 2-AC. Inference open step 3 →

The scaling recipe, ingredient by ingredient

Data. VideoMix22M samples SSv2 (168k clips, 168 h, weight 0.056), Kinetics 400/600/700 (733k, 614 h, 0.188), HowTo100M (1.1M, 134k h, 0.318), curated YT-Temporal-1B (19M, 1.6M h, 0.188) and ImageNet (1M images, repeated into 16 frames, 0.250). Curation splits YT1B into 316M scenes with PySceneDetect, embeds each scene's middle frame with DINOv2 ViT-L, clusters into 1.5M clusters, keeps the 210k clusters that contain at least one Kinetics, SSv2, COIN or Epic-Kitchens training video, and reweights them toward that target mix, leaving 115M scenes. Paper Table 1, App. A.2, Table 11 Appendix A.2 gives YT1B as 1.4M hours and Table 1 as 1.6M. Paper

Schedule and resolution. A 12k-iteration warmup and a 228k-iteration constant phase at 16 frames and 256 px, then a 12k-iteration cooldown at 64 frames and 384 px with the learning rate decayed from 5.25e-4 to 1e-6, at a global batch of 3,072. Training at 64 × 384 × 384 throughout "would require roughly 60 GPU-years"; the progressive schedule cuts GPU time 8.4×. Paper §2.4, Table 9, Fig. 5

Token counts. A pretraining clip is (16/2)·(256/16)² = 8 · 256 = 2,048 tokens. A cooldown clip is (64/2)·(384/16)² = 32 · 576 = 18,432 tokens, 9× more, so the attention term of each layer costs about 81× more. A V-JEPA 2-AC frame is 1 · 256 = 256 tokens, and the VidQA model at 384 px feeds the LLM 576 tokens per 2-frame tubelet, the 288 per frame the paper quotes. computed from Table 9 and §7.4 open step 2 →

The attentive probe, and what it measures

Four transformer blocks with 16 heads sit on the frozen encoder's tokens; the last block replaces self-attention with cross-attention from a single learnable query, and a linear classifier reads the query. Paper §5, App. C.1

A linear probe on pooled features measures what is linearly available in one vector, which undercounts a patch-token model with no class token. Full fine-tuning measures initialisation plus adaptation, and mixes the two. An attentive probe sits between: a small nonlinear reader that may attend anywhere in the token set while the encoder stays fixed. The four-block probe beats a single cross-attention block by 1.4 points on average for ViT-L and 1.0 for ViT-g. App. C.2 Two protocol details matter for comparisons. Each run trains 20 probe heads with different learning rates and weight decays and reports the best, with no separate validation split stated. Jester and Diving-48 read four encoder layers, not one. Table 15, Table 16 open step 7 →

Why the motion columns favour a JEPA. The target of each masked tube is the feature of a region whose content is determined by the trajectory visible around it, so features that encode how things move are the cheapest way to predict it. Image-text contrastive encoders are trained per frame against captions, which name objects far more often than motions. The pattern in Table 4 fits: V-JEPA 2 leads SSv2 by 20 points over the best image encoder and trails SigLIP2 and PE on K400, COIN and ImageNet by 1 to 4. Inference

Action anticipation (Epic-Kitchens-100)

The context clip ends 1 s before an action; the encoder reads 32 frames at 8 fps, the pretraining predictor is given mask tokens for the frame 1 s ahead, and a probe with three queries (action, verb, noun) trained with focal loss reads both. Recall@5 on actions: ViT-L 32.7, ViT-H 36.5, ViT-g 38.0, ViT-g384 39.7, against 27.6 for PlausiVL (8B), 26.0 for Video-LLaMA and 25.8 for InAViT, a 44% relative gain. Paper §6, Table 5, App. D.1 The probe-input ablation says what that gain is made of: encoder features alone 39.1, predictor output alone 20.2, both 39.7. Accuracy falls sharply as the anticipation time grows from 1 s to 10 s. Table 20, Fig. 18 The baselines are task-specific systems with different backbones and data; none is a frozen-encoder probe. Inference

Video question answering

LLaVA-style alignment: encoder tokens, a projector, an LLM, trained in stages (captioning, image QA, video QA). The controlled comparison fixes Qwen2-7B-Instruct, 18M samples, a frozen encoder and 128 frames, and swaps the vision encoder: V-JEPA 2 ViT-g512 averages 52.3 against 49.1 for PE, 48.1 for SigLIP2 and 45.7 for DINOv2, with its largest margins on MVP, TemporalBench and TVBench, and a narrow loss on PerceptionTest (72.0 against 72.4). Paper §7.2, Table 6 The headline row uses Llama 3.1 8B, the full 88.5M samples, ViT-g384 with an MLP projector and 288 tokens per frame: PerceptionTest 84.0, MVP 44.5, TempCompass 76.9, TemporalBench 36.7, TOMATO 40.3, TVBench 60.6, MVBench 73.5, against PerceptionLM 8B's 82.7, 39.7, 72.7, 28.3, 33.2, 63.5 and 77.1. Table 8

Physical-understanding probes

On IntPhys 2 (violation of expectation, chance 50%), V-JEPA 2 ViT-H scores 54.0 easy, 58.5 medium, 59.4 hard, 57.5 overall and 56.4 held-out, against human majority votes of 96.2, 97.8, 95.5, 96.4 and 92.4. It is the best predictive model evaluated, and each model's figure is its best of about a dozen hyperparameter runs per column. On the original IntPhys, where V-JEPA ViT-H with RoPE reaches 98.3, V-JEPA 2 ViT-H reaches 87.2. IntPhys 2, Table 2

The gap is not subtle. A frame-feature objective rewards continuity of appearance, which covers permanence and some continuity, but nothing in it penalises two solids passing through each other. MVP's 44.5 paired accuracy leads the 8B class, by 4.8 points over PerceptionLM, and is still far from a solved benchmark. Inference

What a DROID frame is: the file clock and the wall clock

The loader asks the video file for its frame rate and strides by ⌈rate / 4⌉ frames to get "4 fps". DROID's MP4s do not carry the capture rate. Each holds one frame per 15 Hz control step but is timestamped as if recorded at 60 fps, so the file plays four times too fast and the loader takes every fifteenth frame. Two raw episodes, read on 12 Oct 2026: droid.py at 204698b; DROID raw 1.0.1

What changes. One step of the world model is about a second of teleoperated motion, not a quarter of one. That fits the numbers that looked odd before: the repository's example step moves 12.9 cm, and the planner's box allows up to 13 cm per step, ordinary for a second of motion and fast for 0.27 s. The paper's "videos typically 3–4 seconds long" are their file durations; real DROID episodes run about four times longer. Any comparison that treats V-JEPA 2-AC as a 4 Hz model is off by a factor of four, and models trained at true 5 fps, as in H-JEPA, see a different action granularity. Whether Meta's internal training pipeline read the same tag cannot be checked from outside; the released loader does. Inference

The predictor as a function

Each time step contributes a block of 258 tokens in the order [a_t, s_t, 256 patches]. The attention mask lets every token in block t attend to every token in blocks 0 to t, including the action a_t of its own block, and to nothing later. The patch outputs of block t are read as the prediction of frame t + 1. Patches carry the full 3D rotary position; action and state tokens get only the temporal rotation. ac_predictor.py; modules.py build_action_block_causal_attention_mask §3.1

Teacher forcing plus a short rollout

Teacher forcing predicts every next frame from ground-truth latents. The rollout term feeds the first prediction back in and supervises the second, so the predictor is differentiated through one recurrent step. Eqs. 2 to 4 In code the rollout starts from [z₀, ẑ₁], appends ẑ₂, and the loss is a per-element mean absolute error over both. train.py, auto_steps: 2

Why so short: each rollout step reruns all 24 layers over a growing sequence and keeps the activations for the backward pass, and gradients through recurrent steps compound. What two steps buy is exposure to the model's own one-step error once, which controls the first-order drift term. Nothing constrains error at step five, which is one reason the planner never asks for step five. Inference open step 10 →

Planning as energy minimisation

The current frame and the goal image are encoded by the same frozen encoder. The cross-entropy method samples each action coordinate from a Gaussian initialised at zero mean and unit variance, rolls every sample through the predictor, keeps the top k by energy, refits mean and variance to them, and after the last iteration returns the mean. Deployed settings: 800 samples, top 10 (an elite fraction of 1.25%), 10 iterations, horizon 1, blocking control that waits for each action to finish. §3.2, Fig. 7, §4.1, App. B.2 In the released planner the end-effector state during a rollout is not predicted: it is composed analytically from the previous state and the sampled action, rotations are sampled as zero, and the refit is smoothed with momentum. notebooks/utils/mpc_utils.py open step 11 →

The one energy landscape in the paper sweeps Δx and Δy over ±5 cm for a reach whose true displacement is (0, −10) cm. The minimum sits near (0, −5) cm, and the energy spans 0.40 to 0.44. Fig. 9 That minimum lies on the edge of the swept square: the figure shows the landscape slopes the right way, not that its minimum finds the true action, and the whole landscape varies by about 10% of its level. Inference

Tasks. Reach moves the gripper to the pose in one goal image and ends within 4 cm in all three single-axis tests, with the distance falling at every step. Grasp and reach-with-object use one goal image. Pick-and-place is given two sub-goal images and the final goal: the planner chases the first (object grasped) for 4 steps, the second (object near the target) for 10, and the final for 4. §4.2, Fig. 8, App. B.2 Baselines. Octo starts from octo-base-1.5, trained on Open X-Embodiment, and is fine-tuned on all of DROID with hindsight image goals drawn up to 20 steps ahead, two frames of context and a 4-action horizon; it gets the better of blocking and non-blocking control. Cosmos starts from the 7B continuous-token latent diffusion model and is fine-tuned on DROID with three hand-made fixes (a lower learning rate, no conditioning dropout, noise raised by a factor of e²). §4.1

Where the speed gap comes from. One CEM evaluation of V-JEPA 2-AC at horizon 1 is one forward pass of a 300M predictor over 258 tokens, about 2 · 3×10⁸ · 258 ≈ 1.5×10¹¹ FLOPs. At 800 samples and 10 iterations that is 1.2×10¹⁵ FLOPs per action, which an RTX 4090 at half its roughly 165 TFLOP/s dense BF16 peak finishes in about 15 s, close to the reported 16 s. Cosmos spends 15× the time on a tenth of the samples, about 150× more per sample, because each sample is a multi-step denoising of video latents by a 7B network. V-JEPA 2.1's paper reports the same V-JEPA 2 planner at 3 s per action on an A100; neither paper reconciles the two timings. Inference V-JEPA 2.1 Table 6 open step 12 →

Looking inside. A deterministic ViT-L decoder trained with MSE to map V-JEPA 2 features back to pixels, applied off the shelf to predictor outputs, shows the arm animated and the background kept still; the cup ends slightly low after a rollout, and with an open gripper the cup stays put. App. B.3, Fig. 15 These are single qualitative examples. Inference

Camera sensitivity, measured. For camera positions from about 35° to 85° around the table, the robot makes 201 random x–y moves; for each, the planner infers the one-step action that best explains the next frame. A 2 × 2 least-squares map from inferred to executed actions has condition number ≈ 1.5, so it is close to a rotation; the rotation error grows almost linearly with camera angle, and the mean absolute error is about 1.6 cm on 5 cm moves. Rotating the planner's actions by that map would calibrate it without supervision; the experiments do not. App. B.4, Fig. 16 This is a controllability measurement, and it is the most informative quantitative diagnostic of the world model in the paper. Inference

Without the sub-goal images

Table 2's pick-and-place gives the planner three goal images. HWM (arXiv 2604.03208) ran the experiment the paper does not: one goal image, a Franka arm in one lab, 10 start-goal pairs per object with 5 trials each. Its flat baseline uses the V-JEPA 2-AC architecture retrained on the same data as HWM's low level (about 96 hours of DROID and 30 of RoboSet), and the hierarchy adds a long-horizon predictor in the same latent space with learned macro-actions. Intervals below are 95% Wilson, computed by this page from the percentages and n = 50. HWM §3.1, Table 1, App. A.2

The flat planner's zero is the expected failure of a horizon-1 greedy energy on a task that must first move away from the goal image to grasp the object, and the hierarchy's 70% shows a learned subgoal can replace the human's. Two things are not shown: a second lab, and whether separate latent spaces per level, H-JEPA's change, add anything on a real arm. Inference H-JEPA →

V-JEPA 2-AC predicts one frame ahead and plans one step ahead; pick-and-place works only because a person supplies the intermediate goal images. H-JEPA names the two reasons a single-level JEPA world model struggles further out. Rolling one fine-grained predictor over a long horizon compounds its error and grows the action search space. And one latent space has to serve both short-range dynamics and goal matching, although a goal such as "reach that room" is more abstract than the full state that matches it. Its answer is a stack of JEPAs, each predicting farther ahead in a latent space of its own, trained jointly from pixels. H-JEPA §1

Building the hierarchy

Higher levels are built from the latents of the level below. With stride s_ℓ (how far each window advances) and window w_ℓ (how far it spans), and a:b meaning a, …, b−1:

The state encoder pools a w-step window into one abstract state; the action encoder pools the s action embeddings that carry the system from that state to the next upper-level state. Each level is then an ordinary action-conditioned JEPA in its own space, trained by one-step teacher forcing over a context of c_ℓ states, with SIGReg on the batch of embeddings:

Eq. 2 prints the context as z_<τ; the text describes a causal predictor that sees the c_ℓ preceding states and their actions, which is what the line above writes. SIGReg weight λ is shared by all levels of a model: 0.18 on FourRoom, 0.09 elsewhere, with M = 1024 random projections. §2.1, App. C.1, Table 4

Change the depth, stride and window below. The page recomputes how many level-1 frames one training sample needs and checks the result against the paper's Table 3.

The DROID model, with shapes

What each level keeps: selective abstraction

Two-layer MLP probes on frozen latents, reported as variance-normalised MSE (1 is the marginal mean, 0 is exact recovery), three seeds. In four-level models on Visual AntMaze the ant's joint configuration becomes less recoverable with each level while its position stays recoverable; on FourRoomDistractors the independently moving distractor fades while the agent stays. §3, Fig. 4 The effect tracks the data. Measured as the ratio of spectral centroids of the fastest and slowest state components, environments with a frequency gap below about 2× show little abstraction; AntMaze, Humanoid and FourRoom show it, and the manipulation datasets, Push-T and Cube, have small gaps and do not. §3, Fig. 5, App. F

The mechanism the paper offers is the joint training itself: prediction-error gradients flow into every encoder, so each level keeps what it can predict at its own stride and drops what it cannot. That is a statement about what the objective rewards, demonstrated by probes in three environments, not a guarantee; in tabletop manipulation, where object and gripper move on similar timescales, the hierarchy decomposes time but does not abstract state. Inference

Why diverse scenes need an inverse-dynamics term

On DROID the scene, lighting, camera pose and objects change every episode. Within an episode the background is the most predictable part of the signal, so a purely predictive objective can keep the scene identity and drop the arm. The paper calls this slow-feature collapse and measures it as the within-episode share of latent variance, χ(E) = E_b[tr Var_t(z_t | b)] / tr Var_b,t(z_t), which is 0 at exact collapse. Latent variance is 75% between episodes on DROID against 28% on OGBench Cube. §4.3, App. L.1, L.2

The planner

Encode the current and goal observations at every level, giving z^(ℓ)₀ and g^(ℓ). The top level optimises macro-actions so that its final predicted state lands on its own goal latent:

Each lower level then plans toward the subgoals the level above predicted. It rolls its own model forward, lifts the rollout into the upper space through the upper encoder, and matches K of the upper predictions, the last one weighted 1 and the intermediate ones β:

K = 1 is closed-loop planning, matching only the first subgoal; K = H with β = 0 matches only the terminal prediction; β > 0 also tracks the intermediate ones. Every level optimises by gradient descent through its rollout, not by CEM. §2.2, App. E

Why the hierarchy helps: two levers, separated

Hierarchical planning changes two things at once: it splits the search in time, and it scores progress in a more abstract space. The paper isolates the second by keeping level-1 rollouts and planner settings fixed and changing only the space in which the goal distance is measured, projecting both prediction and goal through the upper encoders. On AntMaze every multilevel model gains from at least one projected cost, and the best projected flat planner beats the separately trained LeWM at every depth. Cost maps show why: level-1 distances to an anchor are nearly constant away from it, so a gradient planner gets no signal until it is close, while upper-level distances rise smoothly along corridors. §4.2, Table 1, Fig. 7 The first lever shows in a different measurement: along held-out expert trajectories, the cost of tracking the next subgoal falls monotonically (−Spearman 0.89 to 1.00 for H-JEPA) where the full-horizon goal cost does not (0.48 to 0.75 for the flat level-1 cost). §4.2, Table 10

The toy below makes both levers visible on a maze. It is schematic: a grid, a greedy planner and hand-made costs, not the paper's models.

Results in simulation, every number the paper tabulates

• Depth helps, then the data run out. Three levels is best on AntMaze (73.3) and FourRoom (96.0); Cube peaks at three (60.0); Push-T collapses at three and four levels because a clip must lie within one short episode, leaving three- and four-level models 58% and 14% of LeWM's training transitions per epoch. §4.1, Table 3
• Distinct spaces add to temporal hierarchy where abstraction emerges. Against HWM at matched implementation, data and depth (upper encoders replaced by identities), H-JEPA reaches nearly twice HWM's success on AntMaze, scores higher on Cube at larger depths, and ties on FourRoom and Push-T. HWM itself beats flat LeWM in all four environments. §4.1, Fig. 6
• Projected costs are not free. On Cube the level-3 and level-4 projections roughly halve success, and on Push-T no cost helps; these are the environments without selective abstraction. App. H, Table 9

DROID: real robot video, offline

DROID has no simulator that runs without ray-tracing hardware, so every DROID number is offline and open-loop: one plan covers a fixed 108-frame window (about 7.2 s) centred on the grasp of 16 hand-picked pick-and-place clips with good lighting and visible objects; nothing is executed or replanned. Plans are scored by Fréchet fidelity, how closely the planned end-effector path follows the expert's, relative to not moving at all:

p and p̃ are the cumulative xyz paths of the planned and expert delta poses and d_F is the discrete Fréchet distance. Each planner seed is averaged over clips, and the median over the 3 × 3 training and planner seeds is reported. The authors calibrate it in simulation: on OGBench Cube no episode succeeds below a fidelity floor, but clearing the floor does not predict success; on RoboCasa CloseDrawer it ranks graded progress with a positive, moderate within-configuration rank correlation. They therefore read a DROID gap as a direction, not a size. §4.3, App. J.1, J.2

The corpus. Raw DROID would make H.264 decoding, not the GPU, set the epoch time over about 5.5 TB. The authors re-encode every camera view to 256 × 256 with libx264 at CRF 18 and repack each trajectory: 127.6 GB for 74,530 episodes, about 45× smaller and 17.9× faster to decode, with trajectories bitwise identical and a median video PSNR of 33.75 dB. The 16 evaluation clips are removed from training. App. J.4, J.1

Limitations, stated and inferred

• Fixed strides. Upper levels advance by a fixed stride; variable-duration segments are named as future work. §6
• Action labels are required on diverse scenes. The inverse-dynamics term is the only Ω(1) barrier against slow-feature collapse; time repulsion alone did not clear the zero-action floor. Action-free video is open. App. J.5, L.6, L.7
• One exterior camera at a time, no wrist view. The authors expect joint multi-view encoding, wrist included, to matter most for precise manipulation. App. J.5
• Goals are images. Language-aligned upper latents are a stated direction. §6
• Deterministic at every level. Each predictor is a point estimate trained with L2, so bimodal outcomes such as a slip average out at every timescale, as they do in V-JEPA 2-AC. Inference
• Small models, simple scenes. The simulated results use a ViT-Tiny with fixed camera and background; whether the selective abstraction survives at V-JEPA 2 scale, or on a frozen V-JEPA 2 encoder, is untested. Inference

The robot numbers, with the intervals the paper leaves out

Every cell in Tables 2 and 3 is k successes in n = 10. The table below adds 95% Wilson intervals, computed by this page. At n = 10, 60% runs from 31% to 83%, and 60% against 80% is not a separable difference. Pooling both labs gives n = 20 per task for V-JEPA 2-AC and Octo, which is the fairest single number the data allow. computed here

Protocol flags

• One model, no seeds. Each method is one trained checkpoint; no training-seed variance is reported anywhere in §4. §4
• The camera was tuned for the method. "We manually tried different camera positions before settling on one that worked well across all of our experiments." The baselines were run from the same placement. §4.3, App. B.4
• Goals are hand-made. Start and goal images, and the sub-goal images for pick-and-place, were composed by the experimenters from the deployment camera. App. B.2, Fig. 14
• Baseline parity is partial. Octo trains on all of DROID, several times the 23k trajectories, but only with goals at most 20 steps ahead, a different goal distribution from the tasks'. Cosmos gets a tenth of the samples because of its cost and runs in one lab only. The world models use blocking control; Octo gets its better mode. Inference from §4.1
• Same arm, same gripper class. DROID is a Franka Panda with a Robotiq gripper, and so are both labs. Embodiment transfer is not tested. §4.1 DROID
• The successor's headline mixes settings. V-JEPA 2.1's "+20 points on Grasp" is 60% for V-JEPA 2 at horizon 1 against 80% for V-JEPA 2.1 at horizon 8; at matched settings the difference is 60% against 70%, one trial in ten. V-JEPA 2.1 Table 6

Which generalisation axes "zero-shot in new labs" actually tests

Four axes for a world model, reported separately

Why a low latent error certifies nothing

Limitations, stated and inferred

• Camera position. The action frame is inferred from the image, often without the robot base in view, so the model's axes rotate with the camera. §4.3, App. B.4
• Long horizons. Autoregressive error grows and the search space grows exponentially with horizon; pick-and-place needs sub-goal images. §4.3
• Image goals only. Language goals are named as future work. §4.3, §9
• A non-convex energy with no uncertainty. CEM with 10 elites can lock onto a wrong basin, and a deterministic predictor offers no signal for when its prediction is off-manifold. Inference
• Slow, blocking, gripper-only control. Each model step is about 1 s of DROID motion, each planned step takes 16 s, a parallel gripper, rigid objects. Contact transients, deformables and dexterous hands are outside what was tested. Inference

Protocol flags for H-JEPA and HWM

• Good practice worth copying. Planner settings are chosen on a separate compute-sweep task set under a fixed 100-TFLOP budget, then frozen and evaluated on 50 new tasks per environment; three paired model and planner seeds; the 16 DROID evaluation clips are removed from training. App. E, J.1
• Simulation with a fixed camera and background. Every success rate in §4.1 and §4.2 comes from four simulators; the real-robot evidence is offline. §4
• An offline metric the authors designed. Fréchet fidelity is a necessary condition on Cube and ranks graded progress on RoboCasa; on DROID the paper reads a gap as a direction, not a size, and so should anyone citing it. App. J.2
• Curated clips. The 16 DROID clips were chosen for good lighting, visible objects and large manipulated objects, and windowed around the grasp. App. J.1
• Baselines at a different rate and on a different set. The frozen-encoder baselines run at their native 1 fps on 15 clips, the H-JEPA models at 5 fps on 16, with intervals over planner seeds rather than training seeds. The comparison supports "far more compute for lower fidelity", not a matched ranking. Fig. 9c Inference
• Unmatched training compute across depths. Depths share an epoch budget; deeper models see fewer clips per epoch, and optimiser steps and training FLOPs are not matched. Push-T's failure at three and four levels is attributed to this, not tested. App. C.1, §4.1
• HWM's real-robot table is one lab, five trials per configuration. 50 trials per object is ten times V-JEPA 2-AC's per-cell count, with intervals reported; the flat baseline is HWM's own retrained model, not the public checkpoint. HWM §3.1, Table 1

What it costs

Pretraining. GPU-hours are not disclosed as a number. The paper puts the full-resolution counterfactual at about 60 GPU-years and the progressive schedule at 8.4× less, which implies about 7 A100-years, some 62,000 GPU-hours, for the 64-frame 384 px ViT-g. derived from §2.4 The released configs run pretraining on 16 nodes of 8 GPUs at 24 clips each and the cooldown on 64 nodes at 6 clips each. configs/train/vitg16

Post-training. Not disclosed; the released config uses 4 nodes of 8 GPUs. Counting a frozen ViT-g forward pass on 16 frames (about 8×10¹² FLOPs) and a 300M predictor trained on 4,128 tokens with a two-step rollout (about 1.1×10¹³), the 94.5k-iteration run at batch 256 comes to roughly 5×10²⁰ FLOPs, on the order of 1,000 A100-hours at 40% utilisation. Caching encoder features removes the first term after one pass. Inference

Inference. 16 s per action on one RTX 4090. A pick-and-place episode of 4 + 10 + 4 = 18 planning steps therefore spends about 4.8 minutes planning; the same episode with Cosmos at 4 minutes per step takes 72 minutes, the "over one hour" the paper reports. §4.2, Table 3

Compute and data across the family

Three experiments a strong reviewer would demand

1. Encoder ablation at the AC stage. The same predictor and 62 hours over V-JEPA 2, DINOv2, SigLIP2 and V-JEPA 2.1, each frozen, plus V-JEPA 2 fine-tuned, three seeds each, evaluated on Reach, Grasp and pick-and-place with 50 trials per cell. This is the direct test of the paper's central bet. Cost: 15 runs at about 1,000 A100-hours is 15,000 GPU-hours, far less with cached features, plus about 30 robot-days. Speculation
2. Rollout-loss horizon sweep. Train with rollout horizons 1, 2, 4 and 8; report open-loop latent error at 1 to 16 steps and closed-loop success at planning horizons 1, 4 and 8. V-JEPA 2.1 found longer planning horizons hurt V-JEPA 2 and helped 2.1, so the interaction is real. Cost: 4 runs, 1,000 to 3,000 A100-hours each as rollout memory grows. Speculation V-JEPA 2.1 §3.3
3. Camera and goal robustness. Five camera positions from 35° to 85°, with and without the App. B.4 rotation calibration, and goal images taken from a different episode; 50 trials per cell. No training required: about 40 hours of planning compute and robot time for 500 episodes. Speculation

Twenty-one prerequisites, each with a mechanical picture and one line of maths; the last seven belong to H-JEPA. Click a card for the picture.

Sixteen steps, each isolating one mechanism, then a summary. Drag a 3D scene to orbit it; click it and scroll, or use + and −, to zoom. The segment bar under each scene highlights, isolates, hides or explodes a component, and hovering a component names it with its tensor shape. The depth switch changes how much each caption says, never the scene. Arrow keys move between steps when the tour has focus. The colour code is the same in every scene.

One continuous run through the tour with a single example. A Franka arm in a lab it has never seen must pick up a cup and put it on a plate, given a photo of the result. Each stop pauses on the step that explains it; open that step in detail, then come back to the story. One premise gets corrected on the way: the paper never does this from the final photo alone.

V-JEPA 2-AC works inside eight assumptions, most stated in the paper and a few implicit in its setup. They hold well enough for a tabletop arm and fail, one after another, as the body gets more hands, more joints, a moving head or a mobile base. The matrix below scores each assumption for five embodiments; tap a cell for the reason. The cards under it are the directions I would fund, each with a first experiment, what would falsify it, and a cost. Everything in this section is my reading: Inference where it follows from the evidence above, Speculation where it goes beyond it.

Fourteen claims that circulate about this family, each with the strongest evidence for it, the protocol behind that evidence, and what would change the verdict. Status is mine, judged on the evidence cited.

Open research questions, most decision-relevant first

1. Does hierarchy survive closed-loop contact? Every H-JEPA success is simulated with a fixed camera, or offline on DROID. The first physical test, on HWM's Franka protocol, decides whether separate latent spaces matter on a real arm at all. Inference
2. What replaces inverse dynamics when there are no actions? Internet video has none, and the paper shows SIGReg and time repulsion are not enough on diverse scenes. Latent actions learned from video, or a joint-in-time regulariser, are the candidates. Inference App. J.5, L.6
3. Where should a level's step end? Fixed strides ignore task structure. In manipulation, contact and gripper events mark the natural boundaries; DROID's evaluation already reads its waypoints off the gripper trace. Inference §6, App. J.1
4. Can a frozen foundation encoder be the bottom level? V-JEPA 2-AC brings a 1B encoder trained on a million hours; H-JEPA brings a training recipe that shapes small encoders per level. Whether the two compose, a frozen V-JEPA 2 or 2.1 at level 1 with learned levels above, is untested. Inference
5. Which latent distance should a planner trust? Both papers plan on a uniform distance over every feature. Tables 1 and 9 show the choice of space changes success by up to a factor of four (FourRoom, two levels: 9.3% native against 38.7% projected); nobody has learned the metric for planning directly. Inference
6. How should uncertainty propagate down the hierarchy? Every predictor in the family is deterministic. A top level that is unsure which of two routes the world takes should hand the level below two subgoals, not their average. Speculation

Data-faithful to the papers or the code: every number in sections 00 to 09 and 14, including the lineage cards, the DROID clock and its measured episodes, the hierarchy builder's recurrence (checked against H-JEPA Table 3), the results explorer (Tables 1 and 9) and the HWM table; the token calculator (step 2); the mask sampler, its statistics and the truncation (3); the shapes and counts in tooltips (4, 7, 9); the EMA equation (5); the ladder values (6); the state and action layouts and the repository's example step (8); the attention-mask structure (9); the CEM algorithm, its defaults and the Fig. 9 minimum and ground-truth markers (11); the sub-goal schedule and timings (12); the audit numbers and intervals (14). Schematic: the pixel-versus-latent toy (1), the grid density in step 2, block sizes (4), the toy linear collapse model (5, a real simulation of a different model), arm, cup and trajectories (8, 12, 13, 16), the latent path and error band (10, from a stated bound with chosen constants), the energy surface shape (11), taxonomy placements (15) and the severity matrix (section 13); the cost-geometry maze in section 07 is a toy that mimics the paper's Fig. 7 qualitatively.

Checked on 12 Oct 2026 in headless Chromium with software WebGL (SwiftShader), at 1440×900 in light and dark mode and at 390×844: sections numbered 00 to 16 in order; all tour steps and the summary visited through the rail; story mode started; the lineage map (13 nodes, cards), the DROID clock (stride 15 at the 60 fps tag, 4 at 15 fps), the HWM interval table, the hierarchy builder (matches H-JEPA Table 3 at one to four levels, flags other strides), the maze (graded and subgoal costs reach the goal, the other two do not), the results explorer (AntMaze 73.3 against 18.0; FourRoom cost ablation 38.7 against 9.3), the ledger filter, the provenance filter including the new H-JEPA level, the accordions, 21 concept cards, 11 direction cards, the assumption matrix and the interval table all exercised; no console or page errors; no horizontal scroll. All in-page anchors resolve; external links answer, except OpenReview and the Meta blog, which refuse scripted requests. Not verified: hover tooltips, drag-to-orbit and wheel zoom with a real pointer, and touch on a physical phone.