How a Latent World Model Works: V-JEPA 2, V-JEPA 2-AC and H-JEPA

Part 01

What a world model is, and why predict in a learned space

A world model maps what the agent has seen and what it is about to do onto what it will see next. Plan with it by asking that question for many candidate actions before moving. The definition used throughout Derived

Formally it is a conditional predictor ŝ_t+1 = f(s_≤t, a_t). The choice that separates the families is what s is. A video model such as Cosmos or Genie predicts pixels, or a compressed code it can decode back to pixels. A JEPA predicts an embedding produced by an encoder that is trained jointly with the predictor, and never decodes.

The argument for the second choice is a decomposition. For any prediction ŷ of a target y given a context x, the expected squared error splits in two:

E {‖ y - \hat{y} ‖}^{2} = \underset{what a model can fix}{\underset{⏟}{{‖ E [y ∣ x] - \hat{y} ‖}^{2}}} + \underset{irreducible}{\underset{⏟}{tr Cov (y ∣ x)}}

The second term is the variance of the target that the context cannot explain.

In pixels, the irreducible term is dominated by things nothing can forecast: leaves, water, sensor noise, specular highlights. A pixel loss still spends gradient on them. In a learned space the encoder is free to drop what it cannot predict, so the loss pays only for what is predictable. The V-JEPA 2 paper makes the same point about "each blade of grass in a field". Paper §1

Lab 1

Predict pixels, or predict features?

A one-dimensional "video": an object moves predictably across 64 pixels, under per-pixel flicker that nothing can forecast. Compare the best possible pixel prediction with a prediction in a feature space that keeps only the object.

Unpredictable flicker

Object speed (px per frame)

Encoder bandwidth

-pixel: fixable

-pixel: irreducible

share wasted

-feature loss

Try

Raise the flicker: the pixel loss grows while nothing about the object changed. That growth is the irreducible term.
Raise the encoder bandwidth: the feature space starts to keep flicker, and its share climbs back toward the pixel value. Lower it below 6: the encoder blurs away the object too, and the share climbs again. A JEPA encoder is rewarded for keeping what moves and dropping what flickers.

What it shows, and what it does not

Why a learned space can spend capacity on the predictable part. The toy's encoder is a fixed low-pass filter; a real JEPA learns its encoder, which is what creates the collapse problem of Part 06. Toy

The energy view

LeCun's 2022 position paper frames this as an energy-based model: a scalar that is low when a continuation y fits a context x, with no need to be a normalised probability. A joint-embedding predictive architecture computes that energy in representation space. LeCun 2022

E (x, y, z) = D (P_{φ} (E_{θ} (x), z), sg (E_{\bar{θ}} (y)))

E_θ: online encoder. E_θ̄: target encoder. P_φ: predictor. z: what the predictor is told about the target (masked positions, or an action). D: a distance (L1 or L2). sg: stop-gradient.

Every model in this lecture is an instance. In V-JEPA 2, x is the visible part of a clip, y the hidden part and z the positions to fill. In V-JEPA 2-AC, x is the past frames, y the next frame and z the action. In H-JEPA there is one such energy per timescale. Planning then searches over z (actions) to make the energy between the predicted future and a goal low.

Part 02

The family on one map

One move repeated: encode the visible, predict the encoding of the hidden or the future. Five design choices change from model to model. The lineage, I-JEPA to H-JEPA Derived

The five choices: what is hidden (image blocks, video tubes, the next frame, a state several steps ahead), what the prediction is conditioned on (positions, actions, pooled macro-actions), the distance (L2 or L1), how collapse is prevented (a slow teacher, a batch regulariser, an action-regression head), and what the result is used for (frozen features, or a simulator a planner searches). Click a node.

Model	Predicts	Anti-collapse	Headline, with its protocol
I-JEPA 2023	4 image blocks from one context block, L2	EMA teacher, stop-gradient	81.1% ImageNet linear probe, ViT-H/16 at 448 px (Table 1)
V-JEPA 2024	masked video tubes, L1	EMA teacher, stop-gradient	81.9 K400, 72.2 SSv2, 77.9 IN1K, frozen ViT-H/16 at 384 px
V-JEPA 2 2025	masked tubes at 1B parameters	EMA fixed at 0.99925	75.3 SSv2 frozen at 256 px, one protocol for every encoder (Table 4)
V-JEPA 2-AC 2025	next-frame latents given action and state	frozen encoder	pick-and-place 80% / 80% (Lab 1) and 80% / 50% (Lab 2) with human sub-goal images, 10 trials per cell
LeWM 2026	next global latent, from pixels	SIGReg only	about 15M parameters, one GPU, hours
HWM 2026	two timescales, one shared space	its backbone's	single-goal Franka pick-and-place 70% / 60% against 0% flat, 50 trials each
H-JEPA Oct 2026	up to four timescales, one space each	SIGReg per level, plus inverse dynamics on DROID	Visual AntMaze 73.3 ± 3.5% (3 levels) against 18.0 ± 3.5% flat, 3 seeds

Which H-JEPA? Three things share the name. LeCun's 2022 proposal of stacked JEPAs (no experiments; restated in Dawid and LeCun, arXiv 2306.02572, §6.2). An unrelated wireless-control paper (Girgis, Labriji and Bennis, arXiv 2602.07000) and several unrelated repositories. And the method this lecture covers: Zhang, Terver, Rabbat, LeCun and Balestriero, arXiv 2610.06805, code at github.com/kevinghst/H-JEPA. Papers

Part 03

From pixels to tokens

A Vision Transformer never sees pixels one by one. It cuts the input into bricks, flattens each brick and projects it to a vector: one token per brick. Patch and tubelet embedding Paper §2.1

For a video x ∈ ℝ^3×T×H×W, V-JEPA 2 uses a 3D convolution whose kernel equals its stride, 2 × 16 × 16. Each brick of 2 frames × 16 × 16 pixels × 3 colours (1,536 numbers) becomes one d-dimensional token; d = 1408 for the 1B ViT-g. The number of tokens is

N = \frac{T}{t_{p}} \cdot \frac{H}{p} \cdot \frac{W}{p}, N = 8 \cdot 16 \cdot 16 = 2048 for 16 frames at 256 px

t_p = 2 frames per tubelet, p = 16 pixels per side. A 64-frame clip at 384 px gives 32 · 24 · 24 = 18,432 tokens.

Attention compares every token with every other, so its cost grows with N², and a 9× longer sequence costs about 81× more per layer. That is why V-JEPA 2 trains at 16 frames and 256 px and only cools down at 64 frames and 384 px. Paper §2.4, Table 9

Lab 2

Tokenize an input

Pick a model's input recipe or set your own. The page draws the brick grid and computes the token count, the tensor shape after embedding, and the cost of one transformer layer.

Frames T

Resolution (px)

Patch size p

Tubelet depth

Width d

-tokens N

-attention entries

-GFLOP per layer

Try

Preset "V-JEPA 2-AC frame": one frame duplicated into a 2-frame tubelet gives 256 tokens, the unit the world model predicts.
Go from the pretraining preset to the cooldown preset and watch the attention entries grow about 81×.

How the cost is computed

Per layer ≈ 24·N·d² + 4·N²·d floating-point operations: the projections and an MLP of width 4d, plus the attention scores and weighted sum. ViT-g's MLP is 6144 wide (4.4d), so this slightly underestimates it. Derived

Part 04

Inside the encoder: attention and 3D rotary positions

Each of ViT-g's 40 blocks lets every token read every other, weighted by how well their queries and keys match, then transforms each token on its own. The transformer block Paper Table 12

h = x + MHSA (LN (x)), y = h + MLP (LN (h)), Attn (Q, K, V) = softmax (\frac{Q K^{⊤}}{\sqrt{d_{h}}}) V

ViT-g: width 1408, 40 blocks, 22 heads of d_h = 64, MLP width 6144. Q, K, V are linear maps of the normalised tokens.

Attention by itself does not know where a token is. V-JEPA 2 tells it with 3D rotary position embedding (RoPE): each head's 64 dimensions are split into three segments of 20, rotated by the token's time, row and column index respectively, and the last 4 are left alone. Rotating queries and keys by angles proportional to position makes their dot product depend on the difference of positions, so attention sees relative offsets. Paper §2.1 modules.py

(\begin{matrix} q_{2 j}^{'} \\ q_{2 j + 1}^{'} \end{matrix}) = (\begin{matrix} \cos p ω_{j} & - \sin p ω_{j} \\ \sin p ω_{j} & \cos p ω_{j} \end{matrix}) (\begin{matrix} q_{2 j} \\ q_{2 j + 1} \end{matrix}), ω_{j} = 10000^{- j / (D / 2)}, ⟨ R_{p} q, R_{p'} k ⟩ = ⟨ q, R_{p' - p} k ⟩

Intended RoPE on one axis segment of size D. p is the token's index along that axis.

Lab 3

Rotary positions, intended and as implemented

Score one query against one key as the key slides away along an axis, starting from two different base positions. Intended RoPE gives the same curve from both starts. The released code tiles its frequencies instead of interleaving them, which its own comment calls a bug kept for compatibility with the trained weights.

Segment size D

Second base position

Random query and key, seed

-max gap between curves

-norm after rotation

What the code does

rotate_queries_or_keys builds sin and cos for D/2 frequencies, repeats them end to end (ω₀…ω_D/2−1, ω₀…) and applies them to adjacent pairs, so the two coordinates of a pair get different angles and the map is not a rotation. src/models/utils/modules.py @ 204698b

Does it matter?

The model was trained this way, so it learned around it; the measured gap shows only that attention is not purely relative in position, which is a property papers often assume. Derived

Part 05

The JEPA objective: hide, encode, predict, compare

Hide part of the input. Encode what is left. Ask a predictor for the encoding of what was hidden, and compare it with what a slowly updated copy of the encoder says the hidden part looks like. I-JEPA, V-JEPA, V-JEPA 2 Papers

I-JEPA does this on images. From one large context block (85 to 100% of the image) it predicts the representations of four target blocks (15 to 20% each, aspect 0.75 to 1.5), removed from the context, with an L2 loss. V-JEPA and V-JEPA 2 do it on video, with blocks that run through the whole clip in time (tubes) and an L1 loss, which V-JEPA found more stable:

\min_{θ, φ, Δ_{y}} \frac{1}{| M |} \sum_{i \in M} {‖ P_{φ} {(Δ_{y}, E_{θ} (x_{C}))}_{i} - sg (LN (E_{\bar{θ}} (x)))_{i} ‖}_{1}, \bar{θ} \leftarrow m \bar{θ} + (1 - m) θ

C: visible (context) tokens; M: masked tokens; Δ_y: a learnable mask token placed at each hidden position; LN: LayerNorm on the targets; m = 0.99925 in V-JEPA 2, fixed. Loss averaged over the short-range and long-range masks.

Why tubes: neighbouring frames are almost identical, so a token hidden at time t but visible at t ± 1 could be copied rather than inferred. Masking the whole time extent removes that shortcut. Because the context still contains every frame, this is spatiotemporal inpainting, not extrapolation; the explicit predict-the-next-frame objective appears only in V-JEPA 2-AC. V-JEPA's ablation: random tubes at 90% masking give 51.5 / 46.4 / 55.6 on K400 / SSv2 / IN1K against 72.9 / 67.4 / 72.8 for multiblock. V-JEPA Table 4

Lab 4

Sample the masks

The page runs a port of each released sampler. V-JEPA 2 draws one block size per batch and a position per clip, takes the union of 8 small blocks (short-range) or 2 large ones (long-range), then truncates every clip's index lists to the batch minimum.

Clips per GPU batch

-masked per clip

-encoder sees

-predicted

Try

Long-range at 24 clips: about 81% masked, but after truncation the encoder sees only a few percent of tokens.
Set the batch to 1: no truncation, so "encoder sees" becomes the complement of "masked".

Source

Logic ported from src/masks/multiseq_multiblock3d.py (block size: h = round(√(n·a)), w = round(√(n/a)), n = ⌊scale·H·W⌋) with Table 9's parameters; statistics over 200 batches. V-JEPA 1 described the same recipe as "about 90%" masking. Code, measured

Part 06

Why it does not collapse

The loss above has a perfect, useless minimum: an encoder that outputs the same vector for every input predicts itself exactly. Something must keep the representation alive. Representation collapse Derived

If E_θ(x) = c for every x, then the predictor outputs c and the loss is zero. V-JEPA and V-JEPA 2 rely on three asymmetries. A stop-gradient on the target branch, so the target is never pulled toward the prediction. An EMA teacher, θ̄ ← mθ̄ + (1 − m)θ, a lagged copy; at m = 0.99925 it averages roughly the last 1/(1 − m) ≈ 1,333 steps. A predictor, which gives the online branch a degree of freedom the target lacks. Tian, Chen and Ganguli (ICML 2021) analyse why this works for linear networks; SimSiam showed the EMA is not needed if stop-gradient and a predictor are kept. V-JEPA 2 reports no rank or spectrum diagnostic. Paper §2.1 Tian et al. 2021

Lab 5

Train a linear JEPA and watch it collapse, or not

An 8-dimensional input with decaying variances, two noisy views, a 4-dimensional linear encoder W, a linear predictor P, an EMA target. The page trains it live with gradient descent and plots the singular values of W: the size of each direction the representation keeps.

Stop-gradient on the target Predictor P

EMA momentum m

Weight decay

-directions alive

-final loss

What to expect

No stop-gradient: every singular value falls to zero within about a thousand steps, and the loss reaches zero.
No predictor: a slower collapse.
Stop-gradient plus predictor, with or without EMA: no collapse; with the default weight decay all four directions stay.
The full recipe at weight decay 0: the weakest direction falls far below the others, in some runs to zero. That is the partial dimensional collapse the literature reports, and weight decay is one of the things holding it off here.

What it shows, and what it does not

The mechanism in the simplest setting where it can be computed: the linear setting of Tian et al. A 1B-parameter ViT is not guaranteed to behave the same, and V-JEPA 2 does not measure it. H-JEPA replaces all three tricks with a regulariser on the batch (Part 13). Toy

Part 07

V-JEPA 2, one training step from input to output

One step takes a batch of 16-frame clips, hides most of each, and updates three networks: the encoder, the predictor and the mask token. The teacher is updated by averaging, not by gradients. Pretraining Paper §2 app/vjepa/train.py

Lab 6

Follow one clip through the network

Step through the stages. Each shows the tensor shape at that point, what happens to it, and the equation or line of code responsible. Counts are for ViT-g, a 16-frame 256 px clip and the short-range mask at 24 clips per GPU.

Where the numbers come from

Shapes from the paper's Tables 9 and 12; LayerNorm on targets, the per-mask average and the context and target counts (511 and 953 at 24 clips per GPU) from the released training loop and sampler. Paper Code, measured

Play it

Every architecture in the family, block by block

The same parts recombine across the family: an encoder, a predictor, a target, and a distance. Pick a model and click any block for its shape, its size and what it computes.

Lab 7

Architecture explorer

Diagrams drawn from each paper's own description and, where public, its code. Arrows carry tensors; the dashed box is what is not trained in that model.

The recipe that made it scale

Ingredient	Change	Average of 6 frozen tasks	Gain
V-JEPA baseline	ViT-L/16, VideoMix2M, 90k iterations	84.2
Data	2M → 22M videos (VideoMix22M)	85.2	+1.0
Model	300M → 1B parameters (ViT-g)	86.7	+1.5
Training length	90k → 252k iterations, warmup-constant-decay	87.5	+0.8
Resolution	256 → 384 px, 16 → 64 frames	88.2	+0.7

Paper Fig. 3; each ingredient on top of the previous, one seed each, so the order matters. Progressive resolution (16 frames at 256 px, then a 12k-step cooldown at 64 frames and 384 px) cuts GPU time 8.4× against training at full resolution throughout, which the paper puts at about 60 GPU-years.

Using the frozen encoder

After pretraining the encoder is frozen. To measure what it knows, a small attentive probe is trained on top: four transformer blocks whose last one replaces self-attention with cross-attention from a learnable query token q to all the encoder's tokens, then a linear classifier: ŷ = W · softmax(qK^⊤/√d) V. Only the probe learns. Under one shared protocol, V-JEPA 2 ViT-g reaches 75.3 on Something-Something v2 at 256 px (77.3 at 384 px) against 49.9 to 55.4 for the best image encoders, because SSv2 is about motion, which a single image cannot show. Paper §5, Table 4

Two figures need care. PerceptionTest 84.0 is accuracy after supervised fine-tuning, while the other rows of its table are zero-shot. And on Epic-Kitchens anticipation (39.7 recall@5), a probe on encoder features alone reaches 39.1; the predictor adds 0.6. Paper Table 8, App. D.2

Part 08

V-JEPA 2-AC: frames, states and actions

The world model learns from other people's robot logs with no rewards and no task labels: a frame, the gripper's pose in that frame, and nothing else. An action is simply how the pose changed between two frames. Post-training data Paper §3.1

The data is 23k trajectories from DROID, a Franka Panda arm with a two-finger gripper teleoperated in many labs, successes and failures alike, seen from the left exterior camera at 256 × 256. Each frame carries a 7-D end-effector state s = (x, y, z, three Euler angles, gripper opening) in the robot's base frame. The action between two frames is

a_{k} = s_{k + 1} ⊖ s_{k} = (p_{k + 1} - p_{k}, euler (R_{k + 1} R_{k}^{⊤}), g_{k + 1} - g_{k})

Position difference, the rotation that takes one orientation to the next (as extrinsic xyz Euler angles), gripper difference. From droid.py, poses_to_diffs.

Lab 8

From two gripper poses to one action

Set the gripper's pose in two consecutive frames. The page computes the 7-D action exactly as the loader does, and shows why the rotation part is computed from rotation matrices rather than by subtracting angles.

Frame k

x, y (cm)

yaw (°)

Frame k + 1

x, y (cm)

yaw (°)

pitch of both (°)

Try

Yaw 170° to −175°: the true turn is 15°, but subtracting angles says −345°. The matrix form gives 15°.
Drag the second position: the translation part is a plain difference, in the robot base frame.

Detail

Euler angles follow scipy's lowercase "xyz" (extrinsic) convention used by the loader: R = R_z(γ)R_y(β)R_x(α). Roll is fixed at zero here to keep the picture readable. droid.py

What a "frame" is: the file clock and the wall clock

The paper samples DROID "at 4 frames per second". The loader gets that rate by asking the video file for its frame rate and taking every ⌈rate/4⌉-th frame. DROID's files do not report their capture rate: each holds one frame per 15 Hz control step but is timestamped as if recorded at 60 fps. So the loader takes every fifteenth frame, and consecutive training frames are about one second apart in real time. Code, measured

Episode, left camera	Video frames	File says	Robot steps	Camera timestamps span	File rate
AUTOLab, 2023-07-07	220	3.67 s	221	15.6 s	60 fps
RAIL, 2023-04-05	251	4.18 s	252	17.8 s	60 fps

Consequences: one step of the world model is about a second of teleoperated motion, which fits the repository's 12.9 cm example step and the planner's 13 cm limit per step. The paper's "videos typically 3–4 seconds long" is their file duration; real episodes run about four times longer, so "under 62 hours" is most likely file time as well. H-JEPA reports the same 60 fps tagging and evaluates the public V-JEPA 2-AC checkpoint "at its native 1 fps". Derived H-JEPA App. J.1

Part 09

V-JEPA 2-AC: the predictor and how it is trained

The encoder stays frozen. All the learning of dynamics happens in a 300M-parameter predictor that reads the past frames, states and actions, and outputs the next frame's feature map. The action-conditioned predictor Paper §3.1 ac_predictor.py

Each frame is encoded on its own (duplicated into a 2-frame tubelet) into 256 tokens of 1408 dimensions, and LayerNormed: z_t ∈ ℝ^256×1408. The predictor projects patches to width 1024, embeds the 7-D action and 7-D state with two separate linear maps, and lays each time step out as a block of 258 tokens: [a_t, s_t, 256 patches]. Twenty-four transformer blocks (width 1024, 16 heads) run with a block-causal mask, the two conditioning tokens are dropped, and the patch outputs of block t are read as the prediction of frame t + 1.

z (B, T·256, 1408) → linear → (B, T, 256, 1024) a, s (B, T, 7) → two linears → (B, T, 1, 1024) each interleave [a_t, s_t, z_t] per step → (B, T·258, 1024) # T = 16 → 4,128 tokens 24 blocks width 1024, 16 heads, block-causal mask; patches use 3D-RoPE, a and s only the time axis drop a, s tokens → (B, T·256, 1024) → LN → linear → (B, T·256, 1408) → LN # ẑ_t+1 for every t at once

Lab 9

Build the block-causal attention mask

Rows are queries, columns are keys. A token may attend to every token in its own time step, including the action of that step, and to all earlier steps; never to later ones. Patches are drawn as a few cells per step to keep the grid readable.

Time steps T

Patch tokens drawn per step

Compare: plain causal (token by token)

-entries allowed

-(T + 1) / 2T

Hover or tap a cell.

Why block-causal

Inside one step, the patches see the action that is about to be applied, so the prediction of frame t + 1 can depend on a_t. Across steps, nothing sees the future, so one forward pass yields all T next-frame predictions without leakage. build_action_block_causal_attention_mask

At the paper's size

T = 16 gives 4,128 tokens and (4,128)² ≈ 17M mask entries, 53% of them allowed. Derived

Two losses: teacher forcing and a short rollout

L_{tf} = \frac{1}{T} \sum_{k = 1}^{T} {‖ P_{φ} ({(a_{t}, s_{t}, z_{t})}_{t \leq k}) - z_{k + 1} ‖}_{1}, L_{roll} = {‖ P_{φ} (a_{1 : T}; s_{1}, z_{1}) - z_{T + 1} ‖}_{1}, L = L_{tf} + L_{roll}

Teacher forcing over T = 15 steps from true latents; the rollout term feeds the model its own prediction for T = 2 steps (Eqs. 2 to 4). The released code averages the per-element absolute error over [ẑ₁, ẑ₂].

Teacher forcing only ever shows the predictor true past states. A planner, however, feeds it its own predictions. The rollout term exposes the predictor once to its own one-step error. The lab shows why that matters and why the deployed planner never asks for more than one step.

Lab 10

How one-step error compounds

Model the rollout error as e_k+1 = ρ·e_k + δ: each step adds a fresh error δ, and the model maps the previous error forward with gain ρ, its sensitivity to its own input off the data. Teacher forcing sees only δ.

One-step error δ

Error gain ρ

Gain after rollout training ρ′

-error at 1 step

-at 8 steps

-at 8, with ρ′

Read the curve

e_k = δ(1 − ρ^k)/(1 − ρ). With ρ > 1 errors grow geometrically, so a model fine at one step can be useless at eight. The rollout term penalises e₂ = δ(1 + ρ), which only a lower ρ can reduce. The markers show the deployed horizon (1), the released planner's (2) and V-JEPA 2.1's longer test (8). Derived

What it does not show

The paper reports no open-loop error curve, so δ and ρ for the real model are unknown; the sliders are for intuition. Toy

Training settings. AdamW, batch 256 clips, weight decay 0.04; learning rate warmed from 7.5e-5 to 4.25e-4 over 4,500 iterations, held for 85,500, decayed to 0 over 4,500. The released DROID config trains on 8-frame clips (7 teacher-forced steps) rather than 16, on 4 nodes of 8 GPUs. Estimated cost of the full run: on the order of 1,000 A100-hours, less with cached encoder features. App. B.1 droid-256px-8f.yaml Derived

Part 10

Planning: energy, the cross-entropy method, model-predictive control

To act, encode the current frame and a goal image, imagine the outcome of many candidate actions with the predictor, keep the ones whose imagined future is closest to the goal, refit, repeat. Execute the best first action, look again, and plan again. Planning with V-JEPA 2-AC Paper §3.2

E ({\hat{a}}_{1 : H}) = {‖ P ({\hat{a}}_{1 : H}; s_{k}, z_{k}) - z_{g} ‖}_{1}, a_{1 : H}^{⋆} = \underset{{\hat{a}}_{1 : H}}{argmin} E, execute a_{1}^{⋆}, observe, repeat

The energy is the mean absolute difference between the predicted feature map H steps ahead and the goal's, over 256 × 1408 numbers. It is a terminal cost, not summed along the way.

The cross-entropy method (CEM) searches that energy without gradients. Keep a Gaussian over action sequences, initialised at zero mean. Each iteration: draw S candidate sequences, roll each through the predictor, keep the K with the lowest energy (the elites), and refit the mean and spread to them. After the last iteration, return the mean. Deployed settings: S = 800, K = 10, 10 iterations, horizon H = 1, each translation axis clipped to ±7.5 cm. The released planner smooths the refit with momentum, μ ← 0.85·μ_elite + 0.15·μ (likewise σ), and samples 400 over a horizon of 2. Paper App. B.2 mpc_utils.py, world_model_wrapper.py

Lab 11

Build a latent world model, then plan with it

A dot in a square arena, seen as a 20 × 20 image. A frozen encoder maps each image to a 12-D latent, standing in for the frozen V-JEPA 2 encoder. Collect random interaction, fit an action-conditioned predictor on it, then click anywhere to set a goal image and watch CEM plan in latent space, one step at a time.

1 · Data and predictor

Random transitions collected

-one-step error

-copy-last error

2 · Planner

Horizon H

Samples S

Elites K

CEM iterations

Camera rotated at deployment (°)

-energy now

-true distance

-result

Click the arena to set a goal.

What is inside

Encoder: 12 fixed smooth random fields; z = R·image. Never trained on actions, like V-JEPA 2's encoder.
Predictor: z′ = z + W·tanh(U[z; a] + b), a 256-unit random hidden layer whose readout W is fitted by ridge regression on the collected transitions.
Planner: CEM on the L1 energy between the predicted latent H steps ahead and the goal latent, with the released code's momentum; MPC executes the first action.

Try

Fit with 100 transitions, then 2,000: watch the one-step error and the success rate.
Rotate the camera at deployment. Up to about 60° the planner still arrives along a curved path; past 90° it fails. V-JEPA 2-AC shows the same kind of camera sensitivity (Part 11).
Run to goal from the default start and watch the shading: the planner follows the energy, the L1 distance in the encoder's space, not the distance in the arena. The straight line crosses a ridge of high energy, so the path bends around it.
Set K = S: CEM degenerates to averaging random actions.

What it shows, and what it does not

The full loop of a latent world model, pixels to plan, small enough to run in your browser. It is not V-JEPA 2-AC: the encoder is not learned, the world has no contact, and the predictor is fitted, not trained by gradient descent. Toy

# V-JEPA 2-AC deployment, as pseudocode (§3.2, App. B.2) z_g = LN(E(goal image)) loop: z_k, s_k = LN(E(current frame)), proprioception μ, σ = 0, 1 for each action coordinate over the horizon repeat 10 times: A = 800 samples ~ N(μ, σ²), each translation axis clipped to ±0.075 energy_i = mean |P(A_i; s_k, z_k) − z_g| # 256 × 1408 numbers μ, σ = mean and std of the 10 lowest-energy samples execute μ, wait for the arm to finish, observe # pick-and-place: sub-goal 1 for 4 steps, 2 for 10, goal for 4

Why it takes 16 seconds. One energy evaluation at horizon 1 is one forward pass of a 300M predictor over 258 tokens, about 2 · 3×10⁸ · 258 ≈ 1.5×10¹¹ operations. 800 samples × 10 iterations makes 1.2×10¹⁵ per action, about 15 s on an RTX 4090 at half its dense peak, close to the reported 16 s. The Cosmos baseline needs 4 minutes per action for a tenth of the samples. Derived Table 3

Part 11

What V-JEPA 2-AC achieved, and where it stops

Zero-shot pick-and-place on Franka arms in two labs absent from DROID, with the same weights and code. Each cell is 10 trials, the multi-stage task needs sub-goal images from a person, and the camera was placed by hand. Paper Tables 2 and 3, §4 Paper

Both labs pooled, n = 20 per task for V-JEPA 2-AC and Octo; Cosmos ran in Lab 2 only (n = 10). Intervals: 95% Wilson, computed here. Derived

From a single goal image

HWM (arXiv 2604.03208) ran the test the paper does not: one goal image, a Franka in one lab, 10 start-goal pairs per object × 5 trials. Its flat baseline is the V-JEPA 2-AC architecture retrained on the same data as HWM's low level; the hierarchy adds a long-horizon predictor in the same latent space. HWM Table 1

Drawer rates (7 tasks × 5 trials) are not multiples of 1/35, so no counts or intervals can be recovered for them. π0-FAST-DROID and π0.5-DROID are policies fine-tuned on DROID.

Three limits the evidence shows

The camera defines the action frame. For camera positions from about 35° to 85°, the action the model infers for 201 random moves is a rotated copy of the executed one, with the rotation error growing almost linearly in the camera angle, about 1.6 cm mean error on 5 cm moves. Lab 11's camera slider reproduces the mechanism. App. B.4
Greedy energy, short horizon. Horizon 1 with a terminal energy cannot plan a task that must first move away from the goal image; that is why pick-and-place needs sub-goals, and why the flat planner scores 0 of 50 in HWM's test. Derived
Small things are a small share of the energy. The energy averages 256 × 1408 numbers equally. A cup covers a handful of the 256 patches, so a near-miss grasp differs from the goal in about 1.6% of the terms, while the whole energy landscape in the paper's Fig. 9 spans only about 10% of its level. Derived

Part 12

H-JEPA: a stack of world models, one per timescale

Each level is an action-conditioned JEPA in its own latent space. Higher levels look further ahead, in coarser time, over states built from the level below. Zhang, Terver, Rabbat, LeCun, Balestriero, arXiv 2610.06805 H-JEPA §2.1

The problem it addresses: a single-level JEPA world model rolled over a long horizon compounds its error (Lab 10) and searches an action space that grows with every step, and one latent space has to serve both short-range dynamics and goal matching, even when the goal ("reach that room") is more abstract than the state that satisfies it. H-JEPA's answer is a hierarchy trained end to end from pixels. Level 1 encodes each observation, z⁽¹⁾_t = E⁽¹⁾(o_t), and each action block, a⁽¹⁾_t = A⁽¹⁾(a_t). Level ℓ > 1 pools the level below with a stride s_ℓ (how far each window advances) and a window w_ℓ (how many steps it spans):

z_{t}^{(ℓ)} = E^{(ℓ)} (z_{t s_{ℓ} : t s_{ℓ} + w_{ℓ}}^{(ℓ - 1)}), a_{t}^{(ℓ)} = A^{(ℓ)} (a_{t s_{ℓ} + w_{ℓ} - 1 : (t + 1) s_{ℓ} + w_{ℓ} - 1}^{(ℓ - 1)})

Eq. 1. The state encoder pools a w-step window into one abstract state; the action encoder pools the s action embeddings that move the system to the next upper-level state (a macro-action). All experiments use w = 1.

Every level is then trained as its own JEPA, by one-step teacher forcing over a context of c_ℓ states, plus a regulariser on its batch of embeddings, and all levels are summed and trained together, with upper-level losses backpropagating into the encoders below:

L^{(ℓ)} = \frac{1}{c_{ℓ}} \sum_{τ = 1}^{c_{ℓ}} {‖ F^{(ℓ)} (z_{1 : τ}^{(ℓ)}, a_{1 : τ}^{(ℓ)}) - z_{τ + 1}^{(ℓ)} ‖}_{2}^{2} + λ_{ℓ} SIGReg (Z^{(ℓ)}) + γ_{ℓ} L_{idm}^{(ℓ)}, L = \sum_{ℓ} L^{(ℓ)}

Eqs. 2, 3 and 8. No EMA target, no stop-gradient on targets, no pretrained weights. The inverse-dynamics term (γ > 0) is used on DROID only. Eq. 2 prints the context as z_<τ; the line above follows the text's causal description.

Simulation models. Level 1 is LeWM: a ViT-Tiny encoder read through its [CLS] token and an MLP projector with BatchNorm, and a causal transformer predictor (4 layers, 8 heads, width 192) conditioned on actions through adaptive LayerNorm; a level-1 step is 5 environment steps. Upper levels: a two-layer MLP encoder on one lower-level latent, a one-layer transformer pooling two action embeddings into a 4-D macro-action, a causal transformer predictor; s = 2 and w = 1, so each level up doubles the time a prediction spans. AdamW, H200 GPUs, three seeds. H-JEPA App. C.1

Lab 12

Build the hierarchy

Choose depth, stride and window. The page lays one training sample out on level-1 time, computes how many level-1 frames it needs and how long a clip must be, and checks the result against the paper's Table 3.

The arithmetic

The top level needs n_L = 4 states; then n_ℓ−1 = (n_ℓ − 1)s_ℓ + w_ℓ, and a clip spans 5(n₁ − 1) + 1 environment steps. To cover K upper-level subgoals, a lower level plans H_ℓ = K·s_ℓ+1 + w_ℓ+1 − 1 steps. App. C.1, §2.2

Why depth costs data

A clip must lie inside one episode. Four levels need 121-step clips, so short-episode datasets such as Push-T yield only 14% of the flat model's training transitions per epoch, and four-level Push-T fails. Table 3, §4.1

# the DROID model (two levels), from Table 5 level 1 frame 256×256, 5 fps → ViT-S/16 → [CLS] → projector → z⁽¹⁾ ∈ ℝ³⁸⁴; action ∈ ℝ⁷ via adaptive LayerNorm predictor: causal transformer, width 384, depth 12, 6 heads, FF 1536; context 7 states loss: prediction + 0.08·SIGReg + 100·IDM level 2 stride 3: z⁽²⁾_t = MLP₇₆₈(z⁽¹⁾_3t); macro-action = MLP(64)+LN(3 × 7 commands) ∈ ℝ⁸, with 0.005·SIGReg predictor width 384, depth 12; context 2 states; loss: prediction + 0.16·SIGReg + 50·IDM train AdamW, batch 256, peak lr 5e-4, 100 epochs × 292 iterations, bf16, three seeds

What each level keeps

Probes on the frozen latents of four-level models show selective abstraction: on Visual AntMaze the ant's joint configuration becomes less recoverable at each level up while its position stays recoverable; on FourRoom the randomly moving distractor fades while the agent stays. The effect appears only when the fast and slow factors in the data move at timescales at least about 2× apart (AntMaze, Humanoid, FourRoom), not in Push-T or Cube. The mechanism is the joint training: each encoder gets gradient from a predictor working at its own stride, so it keeps what is predictable at that stride. §3, Figs. 4 and 5

Part 13

Collapse without a teacher: SIGReg, and why diverse scenes need inverse dynamics

H-JEPA drops the EMA teacher. It keeps embeddings alive by pushing each batch toward an isotropic Gaussian, and on varied robot scenes it must add a head that recovers the action from consecutive states. H-JEPA App. K, L; LeJEPA Papers

SIGReg (Balestriero and LeCun, LeJEPA) tests normality one direction at a time. Project the n embeddings of a batch onto M random unit directions u_m, and for each projection compare its empirical characteristic function with that of a standard normal. By the Cramér–Wold theorem, matching every one-dimensional projection means matching the whole distribution.

SIGReg (Z) = \frac{1}{M} \sum_{m = 1}^{M} T_{EP} (Z u_{m}), T_{EP} (h) = n \int {| {\hat{φ}}_{h} (t) - e^{- t^{2} / 2} |}^{2} w (t) d t, {\hat{φ}}_{h} (t) = \frac{1}{n} \sum_{i} e^{i t h_{i}}

Eq. 9 with the Epps–Pulley statistic; Gaussian weight w(t). The LeWM implementation evaluates the integral on [0, 3] with 17 trapezoidal knots, using that the integrand is even. On Gaussian data the statistic sits at a floor of about 1; on collapsed data it grows linearly with n.

Lab 13

Run SIGReg on a batch of embeddings

Draw n embeddings of dimension 16 from a distribution, project them on M random directions and compute the Epps–Pulley statistic of each projection, the way Eq. 9 does. The histogram shows one bar per direction.

Batch size n

Directions M

-SIGReg

-Gaussian floor

Try

Isotropic Gaussian: about 1 whatever n is.
Collapsed, then double n: the statistic doubles, so the push away from collapse per sample stays constant, the property H-JEPA's App. K.3 relies on.
Low-rank: most random directions see a Gaussian of the wrong width.

What it shows, and what it does not

The statistic and its scaling, computed exactly on the sampled batch with the quadrature described. Its weighting constants may differ in detail from the released code. Toy computed

Slow-feature collapse

On DROID the scene, lighting, camera pose and objects change every episode, and within one episode the background is the most predictable thing in view. A purely predictive encoder can therefore keep the scene identity and drop the moving arm: its latents stay constant within each episode and next-step prediction becomes trivial. The paper calls this slow-feature collapse and measures it by the within-episode share of latent variance, χ, which is 0 at exact collapse. Latent variance is 75% between episodes on DROID against 28% on OGBench Cube. §4.3, App. L.1, L.2

Its key results: no regulariser that looks only at the distribution of single embeddings can prevent it, SIGReg included, because a background-only encoder can have any distribution you like (Proposition 1); the prediction objective actually prefers it, since it has zero irreducible error (Theorem 1); and an inverse-dynamics head, which must recover a_t from (z_t, z_t+1), makes collapse cost at least the action variance that episode identity cannot explain (Proposition 3).

Lab 14

Why the objective prefers the background

A one-feature world: a background b that is constant within an episode, and an agent x that moves with the action plus a random innovation ε. The encoder mixes them, z = cos φ·b + sin φ·x + noise, with the same variance for every φ, so a marginal regulariser cannot tell encoders apart. The plot shows each loss as a function of φ.

Agent innovation σ_ε

Sensor noise σ_η

Inverse-dynamics weight γ

-optimal φ

variance on agent

The formulas

L_pred(φ) = sin²φ·σ_ε² + 2σ_η²: reading the agent pays its innovation. L_idm(φ) = σ_a² − sin²φ·σ_a⁴ / (sin²φ(σ_a² + σ_ε²) + 2σ_η²), with σ_a = 1: at φ = 0 it pays the whole action variance, the floor of Proposition 3. Derived

Try

At γ = 0 the optimum is φ = 0, pure background: collapse. Any γ above σ_ε²·2σ_η² moves it off zero, and the agent's share grows with γ; at the default noise and γ = 50, the paper's DROID weight, it reaches 80%. H-JEPA's runs agree: without the term, every DROID run collapsed; with γ = 50, fidelity reached 30.7%. Table 12 Toy

DROID, level 1, 5 fps	Representation	Planning (Fréchet fidelity)
No SIGReg, any IDM weight	effective rank 1 to 14 of 384	below the do-nothing floor
SIGReg only (five weights)	collapses later at larger λ; action sensitivity below 0.05	−81% final at the largest λ
SIGReg + time repulsion	keeps moving within episodes	never clears the floor
SIGReg + IDM, γ = 50	χ = 0.12, sensitivity 0.28	30.66%

Fig. 17 and Table 12, three seeds, 100 epochs. H-JEPA App. K.1, L.5 to L.7

Part 14

Planning top-down, and what it buys

The top level plans macro-actions toward its own goal latent. Its predicted states become the targets of the level below, which plans finer actions to reach them, down to primitive actions. H-JEPA §2.2 Paper

a^{(L), ⋆} = argmin {‖ {\hat{z}}_{H_{L}}^{(L)} - g^{(L)} ‖}_{2}^{2}, a^{(ℓ), ⋆} = argmin [{‖ {\tilde{z}}_{K} - {\hat{z}}_{K}^{(ℓ + 1), ⋆} ‖}^{2} + β \sum_{i < K} {‖ {\tilde{z}}_{i} - {\hat{z}}_{i}^{(ℓ + 1), ⋆} ‖}^{2}]

Eqs. 4 to 6. z̃_i = E^(ℓ+1)(lower-level rollout at the i-th upper step): the lower plan is lifted into the upper space and compared with the upper level's predictions. K = 1 is closed-loop (track only the first subgoal); β weights intermediate subgoals.

Unlike V-JEPA 2-AC, every level optimises its actions by gradient descent through its own rollout (AdamW, at most 90 steps, early stopping), not by CEM. Upper levels have no natural action bounds, so their search starts from a Gaussian fitted to the last 4,000 training macro-actions and is clipped to their 2nd–98th percentiles. The agent executes two level-1 blocks (10 environment steps) and replans from scratch. App. E

Two things change at once, and the paper separates them. Temporal decomposition: tracking the next subgoal is an easy, short problem; along expert trajectories the subgoal-tracking cost falls monotonically (−Spearman 0.89 to 1.00) where the full-horizon goal cost does not (0.48 to 0.75). A better space to measure progress: keeping level-1 rollouts fixed and only measuring goal distance through the upper encoders raises success on AntMaze and FourRoom; level-1 distances are nearly flat far from the goal, upper-level distances rise smoothly along corridors. §4.2, Tables 1, 10, Fig. 7

Lab 15

Cost geometry and subgoals on a maze

A greedy planner moves one cell at a time downhill in a cost. Compare four costs: one that saturates away from the goal (like a level-1 latent), straight-line distance, a graded geodesic distance (like an upper level), and subgoals along that geodesic tracked with the straight-line cost (temporal decomposition).

What it shows

Why a graded cost helps a local optimiser, and why short subproblems make even a poor local cost work, the two levers of §4.2. Toy

What it does not

The paper's planner is gradient descent in learned spaces, not a grid search, and its costs are learned, not designed. The resemblance to Fig. 7 is qualitative. Derived

Every result the paper tabulates in simulation

Success (%), mean ± SE over three seeds, 50 tasks per environment; planner settings chosen beforehand on a separate task set under a 100-TFLOP budget. Tables 1 and 9. H-JEPA §4, App. E, H

DROID: real robot video, offline

DROID has no usable simulator, so every DROID number is offline and open-loop: one plan covers a 108-frame window (about 7.2 s) centred on the grasp of 16 hand-picked pick-and-place clips, and nothing is executed. Plans are scored by Fréchet fidelity (Part 15). Read from the labels of Fig. 9b: LeWM with inverse dynamics about 34%, HWM about 35%, H-JEPA about 40%, the last at lower planner compute; without inverse dynamics the model collapses. The public V-JEPA 2-AC and JEPA-WM checkpoints, run at their native 1 fps on 15 clips, reach lower fidelity at orders of magnitude more planner compute. §4.3, Fig. 9 read from the figure

Part 15

Evaluating a world model

A low prediction error certifies little. Measure four things separately: how faithful the rollouts are, how much the predictions respond to the action, whether the planner's energy tracks real progress, and whether the task gets done. An evaluation protocol Derived

Axis	Why one number is not enough	A protocol that measures it
Latent fidelity	error is measured in a space the model chose; collapse makes it zero	open-loop k-step error normalised by ‖z_t+k − z_t‖, plus a probe of a physical quantity on ẑ
Controllability	a model can track passive motion while ignoring the action	action sensitivity: change of the rollout when actions are swapped (H-JEPA uses it); an inverse-dynamics probe
Planning quality	the planner exploits wherever the model is optimistically wrong	rank correlation between energy and realised distance to goal over executed plans
Task success	10 trials per cell cannot separate 60% from 80%	three or more seeds, 50 or more episodes per task, Wilson intervals

Fréchet fidelity, H-JEPA's offline metric, compares the whole planned end-effector path with the expert's, relative to not moving at all. It exists because a net-displacement error is blind to the detour that pick-and-place requires.

fidelity = \frac{d_{F} (0, \tilde{p}) - d_{F} (p, \tilde{p})}{d_{F} (0, \tilde{p})}

Eq. 7. p: planned cumulative path; p̃: expert path; 0: staying still; d_F: discrete Fréchet distance. 0 means no better than standing still, 100% the expert path.

Lab 16

Score a planned path

The expert reaches down to an object, lifts it and carries it to a target (side view). Shape the planned path and compare Fréchet fidelity with the net-displacement error earlier work used.

Follows the detour

Progress along the path

Wobble

-Fréchet fidelity

-net displacement error

Try

"Straight to the goal": net displacement error 0, so the old metric calls it perfect; Fréchet fidelity says it missed the object.
Stop halfway along a good path: fidelity counts how far along the expert path the plan got, which is what the paper found it ranks on RoboCasa.

Its limits

On OGBench Cube no episode succeeds below a fidelity floor, but clearing it does not predict success: a grasp either closes or not. Read a fidelity gap as a direction, not a size. App. J.2

What is established, and what is not

Claim	Status	Evidence and protocol
Predicting features beats predicting pixels for frozen video representations	established	V-JEPA's matched ablation; V-JEPA 2 at 75.3 on frozen SSv2 against at most 55.4 for image encoders, one protocol
V-JEPA 2-AC plans zero-shot pick-and-place in unseen labs	partly	with human sub-goal images, 10 trials per cell, one model, camera placed by hand
A flat V-JEPA 2-AC planner solves tasks from one goal image	contradicted	0 of 50 for cup and box pick-and-place in HWM's Franka test
A temporal hierarchy makes single-goal pick-and-place work on a real arm	partly	HWM: 35 / 50 and 30 / 50, one lab
Separate latent spaces per level add to a temporal hierarchy	in simulation	about twice HWM's success on Visual AntMaze; ties on FourRoom and Push-T; three seeds
H-JEPA improves real-robot control	untested	DROID results are offline path fidelity on 16 clips
SIGReg alone prevents collapse	not on diverse scenes	every SIGReg-only DROID run collapses; Propositions 1 and 3 explain why
Video pretraining is what makes V-JEPA 2-AC work	untested	no encoder ablation at the action-conditioned stage
V-JEPA 2-AC's frames are about 1 s apart in real time	measured	DROID files tagged 60 fps for 15 Hz capture; the released loader strides by the tag; H-JEPA App. J.1 agrees

Part 16

Research directions, and the sources

Six experiments I would run next, ordered by what they would teach per unit of cost. Each names what would falsify it. These are proposals, not results. Open

How this maps onto my work

Agentic latent world models: top-down subgoal refinement is latent lookahead across timescales, and HWM's 0% to 70% is the clearest evidence yet that a manipulation planner needs it. Representation: an upper level that keeps only what it can predict at its stride is close to the outcome-sufficient representations I study, and the per-entity probes are the test I would reuse. Dexterous manipulation: whether a hierarchy separates finger joints from object pose decides whether it helps hands at all. None of this is a result yet.

Glossary

Click a term for its one-line mathematics.

Sources

The models

Assran, Bardes, Fan, Garrido et al., V-JEPA 2, arXiv 2506.09985v1, and facebookresearch/vjepa2 at 204698b
Zhang, Terver, Rabbat, LeCun, Balestriero, H-JEPA, arXiv 2610.06805v1, and kevinghst/H-JEPA
I-JEPA; V-JEPA; V-JEPA 2.1

World models and anti-collapse

Background and naming

LeCun, A Path Towards Autonomous Machine Intelligence; Dawid and LeCun, arXiv 2306.02572
Unrelated, same name: Hierarchical JEPA for B5G remote control

Data

DROID raw 1.0.1, two episodes read from the public bucket on 12 Oct 2026

Built by Md Selim Sarowar on 12 Oct 2026. Every number graded Paper was read from the cited paper that day; Code facts were read from facebookresearch/vjepa2 at 204698b or measured on DROID files; the toys run small models in the page and are labelled as such. The full teardown with its sixteen-step 3D tour, story mode and audit is at /notes/v-jepa/teardown/.

How a latent world model works, from pixels to plans.

What a world model is, and why predict in a learned space

Predict pixels, or predict features?

Try

What it shows, and what it does not

The energy view

The family on one map

From pixels to tokens

Tokenize an input

Try

How the cost is computed

Inside the encoder: attention and 3D rotary positions

Rotary positions, intended and as implemented

What the code does

Does it matter?

The JEPA objective: hide, encode, predict, compare

Sample the masks

Try

Source

Why it does not collapse

Train a linear JEPA and watch it collapse, or not

What to expect

What it shows, and what it does not

V-JEPA 2, one training step from input to output

Follow one clip through the network

Where the numbers come from

Play it

Every architecture in the family, block by block

Architecture explorer

The recipe that made it scale

Using the frozen encoder

V-JEPA 2-AC: frames, states and actions

From two gripper poses to one action

Try

Detail

What a "frame" is: the file clock and the wall clock

V-JEPA 2-AC: the predictor and how it is trained

Build the block-causal attention mask

Why block-causal

At the paper's size

Two losses: teacher forcing and a short rollout

How one-step error compounds

Read the curve

What it does not show

Planning: energy, the cross-entropy method, model-predictive control

Build a latent world model, then plan with it

What is inside

Try

What it shows, and what it does not

What V-JEPA 2-AC achieved, and where it stops

From a single goal image

Three limits the evidence shows

H-JEPA: a stack of world models, one per timescale

Build the hierarchy

The arithmetic

Why depth costs data

What each level keeps

Collapse without a teacher: SIGReg, and why diverse scenes need inverse dynamics

Run SIGReg on a batch of embeddings

Try

What it shows, and what it does not

Slow-feature collapse

Why the objective prefers the background

The formulas

Try

Planning top-down, and what it buys

Cost geometry and subgoals on a maze

What it shows

What it does not

Every result the paper tabulates in simulation

DROID: real robot video, offline

Evaluating a world model

Score a planned path

Try

Its limits

What is established, and what is not

Research directions, and the sources

Glossary

Sources

The models