V-JEPA 2 and 2-AC, taken apart

Fourteen premises that circulate about these models, including three that the previous version of this page got wrong. Each is checked against the paper, the code at commit 204698b, or a named primary source. Tags: Paper the V-JEPA 2 paper, Code the official repository or numbers measured by running it, Context another primary source, Inference and Speculation my own reasoning.

The energy view

LeCun's 2022 position paper recasts prediction as an energy-based model: a scalar compatibility E(x, y) that is low when the continuation y fits the context x and high otherwise, with no requirement that it normalise into a density. A joint-embedding predictive architecture computes that energy in representation space. An encoder maps both sides, a predictor carries the context forward under a conditioning variable z, and a distance scores the match. LeCun 2022, §4

The argument against pixel prediction is a decomposition. For any predictor ŷ of y given x, E‖y − ŷ‖² = ‖E[y|x] − ŷ‖² + tr Cov(y|x). The second term is irreducible, and in natural video it is dominated by content nothing can forecast: foliage, water, sensor noise, specular highlights. A likelihood objective still hands out gradient in proportion to per-pixel error, so capacity drifts toward the nuisance. A JEPA pays only for what survives the encoder, and the encoder is free to drop what it cannot predict. The paper states the same intuition with "each blade of grass in a field". Paper §1 Littwin et al. (NeurIPS 2024) prove a matching statement for deep linear networks: latent prediction prioritises features with large regression coefficients, reconstruction prioritises high-variance ones. Littwin et al. 2024

Step 1 of the tour turns the decomposition into a slider. open step 1 →

The bet, and what would falsify it

The stated target is to "learn to understand the world and learn to act largely by observation". The recipe splits the learning in two: action-free pretraining on over a million hours of internet video, then action-conditioned post-training on under 62 hours of one robot's logs. Paper abstract, Fig. 1

The split is the central bet because interaction data is the scarce resource. If the representation learned from watching carries the state a controller needs, a small amount of interaction is enough to attach actions to it. Three results would falsify it. First, an action-conditioned predictor trained on the same 62 hours over an image encoder (DINOv2, as in DINO-WM) or a random one matching V-JEPA 2-AC's success would mean the video pretraining is not what carries the result; the paper runs no such encoder ablation for the AC stage. Second, success that scales with robot hours but not with pretraining scale. Third, a behaviour-cloning policy on the same 62 hours matching it. Octo, trained on all of DROID with hindsight goals, loses, which supports the bet but changes architecture, objective and pretraining at once. Inference

Position enters through a 3D rotary embedding: each head's feature dimension is split into three near-equal segments, rotated by the temporal, height and width index respectively. The paper reports it stabilised training of the largest models, relative to V-JEPA's absolute sin-cos table. Paper §2.1 In code each axis gets 2·⌊⌊d_head/3⌋/2⌋ dimensions, so a 64-dim ViT-g head rotates 20 + 20 + 20 and leaves 4 untouched, and the 32-dim pretraining predictor head rotates 10 + 10 + 10 and leaves 2. src/models/utils/modules.py, RoPEAttention

One pretraining step, with shapes

ViT-g, a 16-frame 256 px clip, the short-range mask, B clips per GPU. |C| and |M| are the context and target counts after the collator's batch-minimum truncation, measured at B = 24.

The objective and the teacher

Eq. 1 of the paper. Δ_y is a learnable mask token placed at each dropped position, the loss is taken only over masked positions, and the teacher weights θ̄ are an exponential moving average of the student's. A fixed m of 0.99925 averages over about 1/(1 − m) ≈ 1,333 steps. Paper Eq. 1, Table 9

Why it does not collapse

The prediction term alone is minimised by a constant encoder, so every JEPA needs a mechanism that keeps the representation from going flat. V-JEPA 2 relies on three: a stop-gradient on the target branch, so the target is never pulled toward the prediction; an EMA teacher, a lagged and non-degenerate copy; and an asymmetric predictor, which gives the online branch a degree of freedom the target lacks. Tian, Chen and Ganguli (ICML 2021) show for linear networks that the predictor comes to share eigenvectors with the representation's correlation matrix and that eigenmodes above a threshold grow rather than decay. SimSiam (Chen and He, CVPR 2021) shows stop-gradient plus a predictor suffice without an EMA. None of this is a guarantee: partial dimensional collapse is common in non-contrastive methods. Tian et al. 2021; Chen and He 2021

What V-JEPA 2 shows about collapse empirically: nothing. The paper asserts that the stop-gradient and EMA "prevent representation collapse" and reports no rank, spectrum or variance diagnostic. Paper §2.1

The contrast is with methods that regularise the batch directly. VICReg hinges each dimension's standard deviation above a floor and penalises off-diagonal covariance. SIGReg, used by LeWM, constrains the batch to an isotropic Gaussian by testing 1-D projections against the normal characteristic function (M = 1024 projections, weight 0.1). It removes the EMA, the stop-gradient and the pretrained encoder and trains a 15M-parameter world model from pixels, at a price: every dimension must carry unit variance. On the 2-D Two-Room task LeWM reaches 87% where the other planners reach 97% to 100%, which its authors attribute to SIGReg in very low-complexity environments. LeWM arXiv 2603.19312, §3 and Fig. 6

Step 5 runs a toy linear JEPA live and lets you remove each mechanism. open step 5 →

Masking, measured

Both masks are unions of rectangular blocks that span the full clip in time, so each is a set of tubes. Block size is drawn once per batch (spatial scale 0.15 or 0.7 of the frame, aspect ratio in [0.75, 1.5]) and block positions per clip. The collator then truncates every clip's context and target index lists to the batch minimum. The truncation keeps the first indices in (t, h, w) order, so a heavily masked clip loses its last time slices from the target, and a lightly masked clip loses its last visible time slices from the context. src/masks/multiseq_multiblock3d.py

Why tubes: adjacent frames are nearly redundant, so a masked token with a visible copy at t ± 1 can be interpolated rather than inferred. V-JEPA's ablation found multiblock beats random-tube masking at 90%, which produced "features of low semantic quality", and beats causal masking from early frames only. V-JEPA §4 Because the context contains frames on both sides of every masked step, this is spatiotemporal inpainting, not extrapolation: V-JEPA 2 pretraining produces a state encoder, and the explicit predict-the-next-frame objective appears only in V-JEPA 2-AC. Inference open step 3 →

The scaling recipe, ingredient by ingredient

Data. VideoMix22M samples SSv2 (168k clips, 168 h, weight 0.056), Kinetics 400/600/700 (733k, 614 h, 0.188), HowTo100M (1.1M, 134k h, 0.318), curated YT-Temporal-1B (19M, 1.6M h, 0.188) and ImageNet (1M images, repeated into 16 frames, 0.250). Curation splits YT1B into 316M scenes with PySceneDetect, embeds each scene's middle frame with DINOv2 ViT-L, clusters into 1.5M clusters, keeps the 210k clusters that contain at least one Kinetics, SSv2, COIN or Epic-Kitchens training video, and reweights them toward that target mix, leaving 115M scenes. Paper Table 1, App. A.2, Table 11 Appendix A.2 gives YT1B as 1.4M hours and Table 1 as 1.6M. Paper

Schedule and resolution. A 12k-iteration warmup and a 228k-iteration constant phase at 16 frames and 256 px, then a 12k-iteration cooldown at 64 frames and 384 px with the learning rate decayed from 5.25e-4 to 1e-6, at a global batch of 3,072. Training at 64 × 384 × 384 throughout "would require roughly 60 GPU-years"; the progressive schedule cuts GPU time 8.4×. Paper §2.4, Table 9, Fig. 5

Token counts. A pretraining clip is (16/2)·(256/16)² = 8 · 256 = 2,048 tokens. A cooldown clip is (64/2)·(384/16)² = 32 · 576 = 18,432 tokens, 9× more, so the attention term of each layer costs about 81× more. A V-JEPA 2-AC frame is 1 · 256 = 256 tokens, and the VidQA model at 384 px feeds the LLM 576 tokens per 2-frame tubelet, the 288 per frame the paper quotes. computed from Table 9 and §7.4 open step 2 →

The attentive probe, and what it measures

Four transformer blocks with 16 heads sit on the frozen encoder's tokens; the last block replaces self-attention with cross-attention from a single learnable query, and a linear classifier reads the query. Paper §5, App. C.1

A linear probe on pooled features measures what is linearly available in one vector, which undercounts a patch-token model with no class token. Full fine-tuning measures initialisation plus adaptation, and mixes the two. An attentive probe sits between: a small nonlinear reader that may attend anywhere in the token set while the encoder stays fixed. The four-block probe beats a single cross-attention block by 1.4 points on average for ViT-L and 1.0 for ViT-g. App. C.2 Two protocol details matter for comparisons. Each run trains 20 probe heads with different learning rates and weight decays and reports the best, with no separate validation split stated. Jester and Diving-48 read four encoder layers, not one. Table 15, Table 16 open step 7 →

Why the motion columns favour a JEPA. The target of each masked tube is the feature of a region whose content is determined by the trajectory visible around it, so features that encode how things move are the cheapest way to predict it. Image-text contrastive encoders are trained per frame against captions, which name objects far more often than motions. The pattern in Table 4 fits: V-JEPA 2 leads SSv2 by 20 points over the best image encoder and trails SigLIP2 and PE on K400, COIN and ImageNet by 1 to 4. Inference

Action anticipation (Epic-Kitchens-100)

The context clip ends 1 s before an action; the encoder reads 32 frames at 8 fps, the pretraining predictor is given mask tokens for the frame 1 s ahead, and a probe with three queries (action, verb, noun) trained with focal loss reads both. Recall@5 on actions: ViT-L 32.7, ViT-H 36.5, ViT-g 38.0, ViT-g384 39.7, against 27.6 for PlausiVL (8B), 26.0 for Video-LLaMA and 25.8 for InAViT, a 44% relative gain. Paper §6, Table 5, App. D.1 The probe-input ablation says what that gain is made of: encoder features alone 39.1, predictor output alone 20.2, both 39.7. Accuracy falls sharply as the anticipation time grows from 1 s to 10 s. Table 20, Fig. 18 The baselines are task-specific systems with different backbones and data; none is a frozen-encoder probe. Inference

Video question answering

LLaVA-style alignment: encoder tokens, a projector, an LLM, trained in stages (captioning, image QA, video QA). The controlled comparison fixes Qwen2-7B-Instruct, 18M samples, a frozen encoder and 128 frames, and swaps the vision encoder: V-JEPA 2 ViT-g512 averages 52.3 against 49.1 for PE, 48.1 for SigLIP2 and 45.7 for DINOv2, with its largest margins on MVP, TemporalBench and TVBench, and a narrow loss on PerceptionTest (72.0 against 72.4). Paper §7.2, Table 6 The headline row uses Llama 3.1 8B, the full 88.5M samples, ViT-g384 with an MLP projector and 288 tokens per frame: PerceptionTest 84.0, MVP 44.5, TempCompass 76.9, TemporalBench 36.7, TOMATO 40.3, TVBench 60.6, MVBench 73.5, against PerceptionLM 8B's 82.7, 39.7, 72.7, 28.3, 33.2, 63.5 and 77.1. Table 8

Physical-understanding probes

On IntPhys 2 (violation of expectation, chance 50%), V-JEPA 2 ViT-H scores 54.0 easy, 58.5 medium, 59.4 hard, 57.5 overall and 56.4 held-out, against human majority votes of 96.2, 97.8, 95.5, 96.4 and 92.4. It is the best predictive model evaluated, and each model's figure is its best of about a dozen hyperparameter runs per column. On the original IntPhys, where V-JEPA ViT-H with RoPE reaches 98.3, V-JEPA 2 ViT-H reaches 87.2. IntPhys 2, Table 2

The gap is not subtle. A frame-feature objective rewards continuity of appearance, which covers permanence and some continuity, but nothing in it penalises two solids passing through each other. MVP's 44.5 paired accuracy leads the 8B class, by 4.8 points over PerceptionLM, and is still far from a solved benchmark. Inference

The predictor as a function

Each time step contributes a block of 258 tokens in the order [a_t, s_t, 256 patches]. The attention mask lets every token in block t attend to every token in blocks 0 to t, including the action a_t of its own block, and to nothing later. The patch outputs of block t are read as the prediction of frame t + 1. Patches carry the full 3D rotary position; action and state tokens get only the temporal rotation. ac_predictor.py; modules.py build_action_block_causal_attention_mask §3.1

Teacher forcing plus a short rollout

Teacher forcing predicts every next frame from ground-truth latents. The rollout term feeds the first prediction back in and supervises the second, so the predictor is differentiated through one recurrent step. Eqs. 2 to 4 In code the rollout starts from [z₀, ẑ₁], appends ẑ₂, and the loss is a per-element mean absolute error over both. train.py, auto_steps: 2

Why so short: each rollout step reruns all 24 layers over a growing sequence and keeps the activations for the backward pass, and gradients through recurrent steps compound. What two steps buy is exposure to the model's own one-step error once, which controls the first-order drift term. Nothing constrains error at step five, which is one reason the planner never asks for step five. Inference open step 10 →

Planning as energy minimisation

The current frame and the goal image are encoded by the same frozen encoder. The cross-entropy method samples each action coordinate from a Gaussian initialised at zero mean and unit variance, rolls every sample through the predictor, keeps the top k by energy, refits mean and variance to them, and after the last iteration returns the mean. Deployed settings: 800 samples, top 10 (an elite fraction of 1.25%), 10 iterations, horizon 1, blocking control that waits for each action to finish. §3.2, Fig. 7, §4.1, App. B.2 In the released planner the end-effector state during a rollout is not predicted: it is composed analytically from the previous state and the sampled action, rotations are sampled as zero, and the refit is smoothed with momentum. notebooks/utils/mpc_utils.py open step 11 →

The one energy landscape in the paper sweeps Δx and Δy over ±5 cm for a reach whose true displacement is (0, −10) cm. The minimum sits near (0, −5) cm, and the energy spans 0.40 to 0.44. Fig. 9 That minimum lies on the edge of the swept square: the figure shows the landscape slopes the right way, not that its minimum finds the true action, and the whole landscape varies by about 10% of its level. Inference

Tasks. Reach moves the gripper to the pose in one goal image and ends within 4 cm in all three single-axis tests, with the distance falling at every step. Grasp and reach-with-object use one goal image. Pick-and-place is given two sub-goal images and the final goal: the planner chases the first (object grasped) for 4 steps, the second (object near the target) for 10, and the final for 4. §4.2, Fig. 8, App. B.2 Baselines. Octo starts from octo-base-1.5, trained on Open X-Embodiment, and is fine-tuned on all of DROID with hindsight image goals drawn up to 20 steps ahead, two frames of context and a 4-action horizon; it gets the better of blocking and non-blocking control. Cosmos starts from the 7B continuous-token latent diffusion model and is fine-tuned on DROID with three hand-made fixes (a lower learning rate, no conditioning dropout, noise raised by a factor of e²). §4.1

Where the speed gap comes from. One CEM evaluation of V-JEPA 2-AC at horizon 1 is one forward pass of a 300M predictor over 258 tokens, about 2 · 3×10⁸ · 258 ≈ 1.5×10¹¹ FLOPs. At 800 samples and 10 iterations that is 1.2×10¹⁵ FLOPs per action, which an RTX 4090 at half its roughly 165 TFLOP/s dense BF16 peak finishes in about 15 s, close to the reported 16 s. Cosmos spends 15× the time on a tenth of the samples, about 150× more per sample, because each sample is a multi-step denoising of video latents by a 7B network. V-JEPA 2.1's paper reports the same V-JEPA 2 planner at 3 s per action on an A100; neither paper reconciles the two timings. Inference V-JEPA 2.1 Table 6 open step 12 →

Looking inside. A deterministic ViT-L decoder trained with MSE to map V-JEPA 2 features back to pixels, applied off the shelf to predictor outputs, shows the arm animated and the background kept still; the cup ends slightly low after a rollout, and with an open gripper the cup stays put. App. B.3, Fig. 15 These are single qualitative examples. Inference

Camera sensitivity, measured. For camera positions from about 35° to 85° around the table, the robot makes 201 random x–y moves; for each, the planner infers the one-step action that best explains the next frame. A 2 × 2 least-squares map from inferred to executed actions has condition number ≈ 1.5, so it is close to a rotation; the rotation error grows almost linearly with camera angle, and the mean absolute error is about 1.6 cm on 5 cm moves. Rotating the planner's actions by that map would calibrate it without supervision; the experiments do not. App. B.4, Fig. 16 This is a controllability measurement, and it is the most informative quantitative diagnostic of the world model in the paper. Inference

The robot numbers, with the intervals the paper leaves out

Every cell in Tables 2 and 3 is k successes in n = 10. The table below adds 95% Wilson intervals, computed by this page. At n = 10, 60% runs from 31% to 83%, and 60% against 80% is not a separable difference. Pooling both labs gives n = 20 per task for V-JEPA 2-AC and Octo, which is the fairest single number the data allow. computed here

Protocol flags

• One model, no seeds. Each method is one trained checkpoint; no training-seed variance is reported anywhere in §4. §4
• The camera was tuned for the method. "We manually tried different camera positions before settling on one that worked well across all of our experiments." The baselines were run from the same placement. §4.3, App. B.4
• Goals are hand-made. Start and goal images, and the sub-goal images for pick-and-place, were composed by the experimenters from the deployment camera. App. B.2, Fig. 14
• Baseline parity is partial. Octo trains on all of DROID, several times the 23k trajectories, but only with goals at most 20 steps ahead, a different goal distribution from the tasks'. Cosmos gets a tenth of the samples because of its cost and runs in one lab only. The world models use blocking control; Octo gets its better mode. Inference from §4.1
• Same arm, same gripper class. DROID is a Franka Panda with a Robotiq gripper, and so are both labs. Embodiment transfer is not tested. §4.1 DROID
• The successor's headline mixes settings. V-JEPA 2.1's "+20 points on Grasp" is 60% for V-JEPA 2 at horizon 1 against 80% for V-JEPA 2.1 at horizon 8; at matched settings the difference is 60% against 70%, one trial in ten. V-JEPA 2.1 Table 6

Which generalisation axes "zero-shot in new labs" actually tests

Four axes for a world model, reported separately

Why a low latent error certifies nothing

Limitations, stated and inferred

• Camera position. The action frame is inferred from the image, often without the robot base in view, so the model's axes rotate with the camera. §4.3, App. B.4
• Long horizons. Autoregressive error grows and the search space grows exponentially with horizon; pick-and-place needs sub-goal images. §4.3
• Image goals only. Language goals are named as future work. §4.3, §9
• A non-convex energy with no uncertainty. CEM with 10 elites can lock onto a wrong basin, and a deterministic predictor offers no signal for when its prediction is off-manifold. Inference
• Slow, blocking, gripper-only control. About 4 Hz at best, 16 s of planning per step, a parallel gripper, rigid objects. Contact transients, deformables and dexterous hands are outside what was tested. Inference

What it costs

Pretraining. GPU-hours are not disclosed as a number. The paper puts the full-resolution counterfactual at about 60 GPU-years and the progressive schedule at 8.4× less, which implies about 7 A100-years, some 62,000 GPU-hours, for the 64-frame 384 px ViT-g. derived from §2.4 The released configs run pretraining on 16 nodes of 8 GPUs at 24 clips each and the cooldown on 64 nodes at 6 clips each. configs/train/vitg16

Post-training. Not disclosed; the released config uses 4 nodes of 8 GPUs. Counting a frozen ViT-g forward pass on 16 frames (about 8×10¹² FLOPs) and a 300M predictor trained on 4,128 tokens with a two-step rollout (about 1.1×10¹³), the 94.5k-iteration run at batch 256 comes to roughly 5×10²⁰ FLOPs, on the order of 1,000 A100-hours at 40% utilisation. Caching encoder features removes the first term after one pass. Inference

Inference. 16 s per action on one RTX 4090. A pick-and-place episode of 4 + 10 + 4 = 18 planning steps therefore spends about 4.8 minutes planning; the same episode with Cosmos at 4 minutes per step takes 72 minutes, the "over one hour" the paper reports. §4.2, Table 3

Three experiments a strong reviewer would demand

1. Encoder ablation at the AC stage. The same predictor and 62 hours over V-JEPA 2, DINOv2, SigLIP2 and V-JEPA 2.1, each frozen, plus V-JEPA 2 fine-tuned, three seeds each, evaluated on Reach, Grasp and pick-and-place with 50 trials per cell. This is the direct test of the paper's central bet. Cost: 15 runs at about 1,000 A100-hours is 15,000 GPU-hours, far less with cached features, plus about 30 robot-days. Speculation
2. Rollout-loss horizon sweep. Train with rollout horizons 1, 2, 4 and 8; report open-loop latent error at 1 to 16 steps and closed-loop success at planning horizons 1, 4 and 8. V-JEPA 2.1 found longer planning horizons hurt V-JEPA 2 and helped 2.1, so the interaction is real. Cost: 4 runs, 1,000 to 3,000 A100-hours each as rollout memory grows. Speculation V-JEPA 2.1 §3.3
3. Camera and goal robustness. Five camera positions from 35° to 85°, with and without the App. B.4 rotation calibration, and goal images taken from a different episode; 50 trials per cell. No training required: about 40 hours of planning compute and robot time for 500 episodes. Speculation

Fourteen prerequisites, each with a mechanical picture and one line of maths. Click a card for the picture.

Sixteen steps, each isolating one mechanism, then a summary. Drag a 3D scene to orbit it; click it and scroll, or use + and −, to zoom. The segment bar under each scene highlights, isolates, hides or explodes a component, and hovering a component names it with its tensor shape. The depth switch changes how much each caption says, never the scene. Arrow keys move between steps when the tour has focus. The colour code is the same in every scene.

One continuous run through the tour with a single example. A Franka arm in a lab it has never seen must pick up a cup and put it on a plate, given a photo of the result. Each stop pauses on the step that explains it; open that step in detail, then come back to the story. One premise gets corrected on the way: the paper never does this from the final photo alone.

V-JEPA 2-AC works inside eight assumptions, most stated in the paper and a few implicit in its setup. They hold well enough for a tabletop arm and fail, one after another, as the body gets more hands, more joints, a moving head or a mobile base. The matrix below scores each assumption for five embodiments; tap a cell for the reason. The cards under it are the directions I would fund, each with a first experiment, what would falsify it, and a cost. Everything in this section is my reading: Inference where it follows from the evidence above, Speculation where it goes beyond it.

Data-faithful to the paper or the code: every number in sections 00 to 06; the token calculator (step 2); the mask sampler, its statistics and the truncation (3); the shapes and counts in tooltips (4, 7, 9); the EMA equation (5); the ladder values (6); the state and action layouts and the repository's example step (8); the attention-mask structure (9); the CEM algorithm, its defaults and the Fig. 9 minimum and ground-truth markers (11); the sub-goal schedule and timings (12); the audit numbers and intervals (14). Schematic: the pixel-versus-latent toy (1), the grid density in step 2, block sizes (4), the toy linear collapse model (5, a real simulation of a different model), arm, cup and trajectories (8, 12, 13, 16), the latent path and error band (10, from a stated bound with chosen constants), the energy surface shape (11), taxonomy placements (15) and the severity matrix (10).

Checked on 22 Sep 2026 in headless Chromium with software WebGL (SwiftShader), at 1440×900 in light mode, at 1440×900 in dark mode with reduced motion, and at 390×844: all sixteen steps and the summary render; every slider, select, checkbox, segment button, zoom button and segment chip on every step was exercised, as were the provenance filter, concept cards, audit filters, taxonomy, matrix cells and direction filters; story mode ran all fifteen stops; no console or page errors; no horizontal scroll at any width. Not verified: hover tooltips, drag-to-orbit and wheel zoom with a real pointer, and touch orbit on a physical phone.