Teardown · World-action models · 2026-09-22

Cosmos 3, taken apart

NVIDIA's Cosmos 3 puts a vision-language model, a video and audio generator, a forward and inverse dynamics model and a robot policy into one set of weights. This page walks the mechanism token by token, audits every headline number against its protocol, and ends in a fourteen-step 3D tour you can take apart yourself.

Primary source: arXiv 2606.02800v4, 23 Jun 2026 Weights released 31 May 2026 under OpenMDW-1.1 Reading: 35 min, tour: 15 min

Open the interactive tour Story mode Evaluation audit Missing experiments

Corrections before the answers

The brief that prompted this page carried several premises that the report contradicts or does not support. Each is corrected here once and then used in its corrected form everywhere below.

The two towers are trained in stages, not jointly. The reasoner is pre-trained and fine-tuned first with next-token prediction; its weights then initialise the generator tower, and generator pre-training updates only the generation parameters while the reasoner stays frozen. Report §4, §4.2.1
6D to SO(3) uses an SVD, not Gram-Schmidt. Zhou et al. (2019) proposed Gram-Schmidt for the 6D representation; Cosmos 3 states that it converts the predicted 6D rotation to a 3×3 rotation "using singular value decomposition". Report §2.1.3
The DROID policy does not use the unified 9D pose action. Cosmos3-Nano-Policy-DROID predicts 32 future absolute joint positions at 15 Hz through a freshly initialised action encoder and decoder, with proprioception added. The 9D pose interface is the mid-training and dynamics interface. Report §4.2.5
16B and 64B are two towers, not two dense models. Nano is two copies of an 8B dense transformer (Qwen3-VL-8B), Super two copies of a 32B one (Qwen3-VL-32B). A third variant, Edge (4B, two 2B towers trained from scratch), exists and was released on 20 Jul 2026. Report §2.5 Card
Only one loss weight between modalities is disclosed. The mid-training loss is a sum of per-modality velocity MSEs with modality-specific scales; the action term is scaled by 10×, and the other scales are not disclosed. Report §4.2.2
There is no single action chunk length. The base models predict actions at 10 to 30 Hz over 16 to 400 frames; the DROID forward-dynamics study uses 16-step chunks; the DROID policy uses 32 steps at 15 Hz. Report §6.3.1, §6.2.5, §4.2.5
Seven-camera driving video has no dedicated multi-view mechanism. Synchronised views are concatenated into one canvas and the layout is described in text inside the JSON prompt; the reported driving transfer benchmark is single-view. Report §3.2.3, App. B.5, §6.2.4
"Artificial Analysis arena" results are open-weight rankings of post-trained checkpoints, and the text-to-image entry was a system. Cosmos3-Super-Text2Image ranked #1 among open-weight models and #4 overall inside a test-time loop that used GPT-5.5 to rewrite prompts and Gemini 3.1 Pro as critic. Report §6.2.1, App. B.7
VANTAGE-Bench and TAR are reasoner benchmarks. They score the vision-language tower on smart-infrastructure video, not generation, and both were built by NVIDIA. Report §6.1
The report has no limitations section. Section 8 is a conclusion only; the limitations statement lives in the model cards. Report §8 Card

Strategic position and problem formulation

What NVIDIA means by a world foundation model and by physical AI

The Cosmos line defined a world foundation model in its first paper as a general-purpose world model that is fine-tuned into customised world models downstream, where a world model predicts future observations from past observations and perturbations. The Cosmos 3 report reframes the target: physical AI agents "perceive, reason, and take actions", and they need two coupled capabilities, understanding (inferring latent state, semantics and dynamics from partial observations) and generation (simulating plausible futures and the actions that should follow). Report §1 Cosmos 1 definition: arXiv 2501.03575

Cosmos 1 and 2 shipped these as separate model families: Cosmos-Predict for future video, Cosmos-Transfer for control-conditioned video with one ControlNet branch per control modality, and Cosmos-Reason for vision-language reasoning. Cosmos 3 collapses the three into one Mixture-of-Transformers whose behaviour is selected by how the input tokens are arranged, not by which checkpoint is loaded. The report benchmarks it directly against its own predecessors: Cosmos-Reason2 on the 48 reasoner benchmarks, Cosmos-Predict2.5 on PAIBench-G and Cosmos HUE, Cosmos-Transfer2.5 on PAIBench-C and AVBench-C. Report §2.2, §6.1, §6.2.2, §6.2.4

Why each standalone model class falls short

Write the three model classes as the conditionals they fit. A vision-language model fits p(text | o, l): it can say that the mug will fall, but it has no output space for an action and no generative model of what the pixels do next. A video diffusion model fits p(o_t+1:T | o_≤t, l) on logged video, which marginalises over whatever policy produced the data, Σ_a p(o′ | o, a) π_data(a | o); with no action variable it cannot answer the interventional question of what happens under do(a). A VLA policy fits p(a_t:t+H | o_t, l) by behaviour cloning: it acts, but it has no forward model with which to check a consequence before committing to it. Inference

The report makes the same argument in its own words: video-only training "does not expose the model to controllable interventions: the same initial observation may evolve differently under different robot commands", which is why paired text-video-action data enters at mid-training, and why failed episodes are kept so that the model sees "off-nominal action effects". Report §3.2.3

Action tokens are necessary for causal prediction, not sufficient. The forward-dynamics mode still fits an observational conditional p(o′ | o, a) from logged trajectories. Wherever a hidden variable drives both the recorded action and the outcome (an object's mass that the operator could feel), that conditional differs from the interventional one. Nothing in the report tests the difference. Inference

Targets, and where the open weights fit the business

The report names three uses: synthetic data generation, task- and embodiment-specific specialisation by post-training, and, longer term, training environments for agents (Fig. 2). The application domains are the ones the benchmark sections are organised around: robotics, autonomous vehicles and smart infrastructure. Code, checkpoints, five synthetic datasets and the Cosmos-HUE benchmark are released under the Linux Foundation's OpenMDW-1.1 license. Report §1, §6

The model cards make the hardware story explicit: Super is recommended for H200, B200 and GB200, Nano for RTX Pro 6000, H100 and B200, Edge for Jetson AGX Orin and Thor. Serving paths are vLLM-Omni, vLLM, TensorRT-LLM, SGLang and NIM containers. NVIDIA/cosmos README

The weights are free and the GPU time is not. On the repository's own benchmark table, Nano needs 114.9 s on a B200 to produce a 189-frame 720p clip (about 7.9 s of video), and Super needs 407.5 s. Open weights enlarge the pool of users whose next bottleneck is inference hardware, and the Cosmos Coalition (Agile Robots, Black Forest Labs, Generalist, LTX, Runway and Skild AI) trains on NVIDIA DGX Cloud. The license is the adoption lever; the accelerators are the revenue. inference_benchmarks.md Inference

The newsroom claims Cosmos 3 is "the world's first fully open omnimodel", has "leading physics accuracy", cuts training and evaluation cycles "from months to days", and that Nano delivers "video and action reasoning in fractions of a second". None of these appears in the report, and the last conflicts with the measured generation latencies above unless it refers only to the 4-step policy, whose latency is not published. NVIDIA newsroom, 1 Jun 2026

Architecture and mechanism

The Mixture-of-Transformers layer

Every decoder layer holds two complete parameter sets, one per tower: its own layer norms, its own attention projection matrices and its own feed-forward network. Tokens in the autoregressive (AR) subsequence, meaning language and ViT-encoded vision, go to the reasoner tower; tokens in the diffusion (DM) subsequence, meaning VAE video and image latents, audio and action, go to the generator tower. Both towers are initialised from the same pre-trained Qwen3-VL. Routing is by position in the sequence, not by a learned router, so this is not a mixture of experts in the gating sense. Report §2.3.1 no router: Inference

The towers meet only inside attention. With queries, keys and values computed by each token's own tower:

O_AR = Attn_causal(Q_AR, K_AR, V_AR) (Eq. 7) O_DM = Attn_full(Q_DM, [K_AR; K_DM], [V_AR; V_DM]) (Eq. 8)

Written as one mask over a sequence of n_a AR tokens followed by n_d DM tokens, with 1 meaning "may attend":

M = [ L_{n_a} 0_{n_a×n_d} ] L = lower-triangular ones (causal)
[ 1_{n_d×n_a} 1_{n_d×n_d} ] AR rows never see DM columns; DM rows see everything

Three consequences follow directly. The AR keys and values do not depend on the noisy tokens, so they are computed once and cached across every denoising step ("reasoner tower caching"). The reasoner can never read the generator's output within a pass. And clean conditioning tokens sit inside the DM block, so they attend to the noisy targets as well: the image being animated is re-encoded at every step, not cached. Report §2.3.2, §5.3.1 third point: Inference from Eq. 8

The released configuration adds details the report does not state: QK-normalisation in both towers, grouped-query attention with 8 key-value heads, SiLU-gated feed-forward layers, a rotary base of 5×10⁶, and video_temporal_causal: false, so the generator attends across the whole clip in both temporal directions. Cosmos3-Nano transformer/config.json

How the two objectives are trained

The reasoner is trained with next-token prediction: 22.0M pre-training samples for two epochs with sequences of at most 16k tokens, then 2.2M supervised fine-tuning samples for 8,200 iterations at batch size 512. Those weights initialise the generator tower, and generator pre-training freezes the reasoner. Whether the reasoner stays frozen through mid-training and post-training is not restated. Report §4.1, §4.2.1

The generator uses rectified flow matching for every modality. A noisy latent is the straight-line interpolation between the clean target and Gaussian noise, and the network regresses the constant velocity along that line, with conditioning tokens gated out of the loss:

x_σ = σ·ε + (1 − σ)·x₀, ε ~ N(0, I), v* = ε − x₀, L = Σ_m λ_m · E ‖ M_m ⊙ (v_θ(x_σ, σ, c) − v*) ‖² with λ_action = 10

The noise level σ is drawn independently for each modality in a sample. Images, audio and actions use a logit-normal distribution, videos use "mode sampling" (after Waver), and a shift reparameterisation σ = s·t̄ / (1 + (s − 1)·t̄) with t̄ = 1 − t pushes mass toward high noise. The shift grows with resolution: s = 1, 3, 5 at 256p, 480p and 720p in pre-training, raised to 3, 5 and 10 in mid-training. Actions inherit the vision schedule. The logit-normal and mode-sampling parameters are not disclosed. Text is dropped with probability 0.1 so that classifier-free guidance works at inference. Report §4.2, §4.2.1, §4.2.2

The report never names the ODE solver. The released Diffusers pipeline integrates with a second-order UniPC multistep scheduler in flow-prediction mode, not a plain Euler step; the tour's Euler update is the simplest correct illustration of the same field. scheduler_config.json, pipeline_cosmos3_omni.py

Scale: Nano's generator saw 31.05T tokens in pre-training on 1,024 GB200 GPUs and 2.4T in mid-training; Super saw 17.86T and 1.9T on 2,048 GB200s. Sequences are packed into a fixed 74,000-token context. Steady-state model FLOPs utilisation is 0.23 for Nano and 0.30 for Super. Report §4.2.1, §4.2.2, Tab. 8

An internal inconsistency. The text of §5.2.8 says Nano and Super were benchmarked on 1,024 and 2,048 GPUs; the caption of Table 8 says 2,048 and 4,096. Report §5.2.8, Tab. 8

Tokenisation, modality by modality

Modality	Encoder	Compression	Token width	Tower
Text	Qwen3-VL tokenizer, vocab 151,936	BPE	d_model	AR
Vision, understanding	ViT, 16×16 patches, 2×2 MLP merge, DeepStack features, text timestamps; trained	32×32 px per token	d_model	AR
Vision, generation	Wan2.2-TI2V-5B causal VAE, frozen	4× time, 16×16 space, then 2×2 patch = 32×32	48 ch × 2×2 = 192 → linear	DM
Audio	Audio VAE (Lee et al. 2025b), 48 kHz stereo, frozen	hop 1,920 → 25 tokens/s	64 → linear	DM
Action	Per-embodiment linear W_in^(k)	one token per sample	d^(k) ≤ 64 → d_model	DM

Report §2.1.1 to §2.1.3; token widths, 48 channels and the 64-wide action pad from transformer/config.json and vae/config.json.

A worked count for the 5-second, 720p clip the brief asked about, using the rounding the released pipeline applies. The VAE is causal: it encodes one prime frame and then groups of four, so a clip must be 4k + 1 frames long and T′ = (T − 1)/4 + 1. At 720p the frame is 1280×720, which the VAE turns into 80×45 latents; the 2×2 patchify rounds the odd height up, giving 40×23 tokens per latent frame. pipeline_cosmos3_omni.py: ceil(latent_h / 2)

Quantity	5 s at 24 fps, 720p	Default 189 frames, 720p	Rule
Pixel frames T	121	189	4k + 1
Latent frames T′	31	48	(T − 1)/4 + 1
VAE latent grid	31 × 45 × 80 × 48	48 × 45 × 80 × 48	÷16 space
Tokens per latent frame	920	920	⌈45/2⌉ × 80/2
Video tokens	28,520	44,160	T′ × 920
Audio tokens	127	197	⌈T/fps × 25⌉
Fits the 74,000-token context	yes	yes	720p capped at 300 frames (69,000)

The understanding path is far more expensive per frame for the same pixels, 900 ViT tokens per 720p frame before any temporal merging, which is why reasoner pre-training caps a sample at 8,192 video tokens. The ViT's frame sampling rate for video is not disclosed. Report §4.1.1 Inference

Multimodal position embedding

Positions are a 3D multimodal RoPE with coordinates (t, h, w). Language tokens use t = h = w, which reduces to 1D RoPE. ViT tokens share t within a frame and vary h, w, exactly as in Qwen3-VL. VAE video tokens vary on all three axes, with spatial indices reset at the start of each vision segment. Audio and action tokens carry time only, h = w = 0. Every DM token is shifted by a fixed temporal gap of 15,000 after the last AR token, added because adjacent positions for the last word and the first frame produced over-saturation and checkerboard artefacts, worst in Super. Report §2.4.1

Sequence order is blocked by modality: all AR tokens, then clean conditioning tokens ordered vision, audio, action, then noisy targets in the same order. Time alignment therefore comes from the temporal coordinate, not from interleaving. Absolute temporal modulation sets the step between consecutive tokens of a stream to δt = TPS_base / TPS with TPS_base = 24/4 = 6 (Eq. 9). A 24 fps video advances 6 units per second, a 30 fps one also 6 (δt = 0.8), audio at 25 tokens/s advances δt = 0.24 per token and again 6 per second, and a 15 Hz action stream advances δt = 0.4. One second of the physical world costs six position units in every modality. Report §2.2.1, §2.4.2 arithmetic

The released config splits the 64 rotary frequency pairs of each 128-wide head into 24 temporal, 20 height and 20 width, interleaved. rope_scaling.mrope_section = [24, 20, 20]

The modulation was ablated. Across four frame-rate bands, three seeds and roughly 400 prompts, text-only FPS control lifts a motion-fidelity composite from 8.51 to 9.28, RoPE modulation alone to 9.63, and both together to 9.81. Report App. E.2, Tab. 29

Nano against Super

Variant	Layers	Hidden	Heads	KV heads	Head dim	FFN	Dense tower	Total	Recommended hardware
Edge	28	2,048	16	8	128	9,216	2B, from scratch	4B	Jetson AGX Orin, Thor, RTX Pro 6000
Nano	36	4,096	32	8	128	12,288	8B, Qwen3-VL-8B	16B	RTX Pro 6000, H100, B200
Super	64	5,120	64	8	128	25,600	32B, Qwen3-VL-32B	64B	H200, B200, GB200

Report §2.5, Tab. 2; hardware from the NVIDIA/cosmos README.

By construction the split between reasoner and generator is even, about 8B and 8B for Nano. How the shared ViT, the embeddings and the modality projections are counted in the headline totals is not disclosed. In BF16 the weights alone are about 32 GB for Nano and 128 GB for Super, which is why the README shards Super across four GPUs and why the DROID policy is served on two RTX Pro 6000 cards. arithmetic Report §4.2.5

Action as a first-class modality

One interface for six embodiments

An action is built from at most three parts: an ego pose for the agent's main observation frame, effector poses, and grasp states. Ego and effector poses are pseudo-actions, the relative transform between consecutive SE(3) poses, ΔT_t = T_t−1⁻¹ T_t, written as a 3D translation plus the 6D rotation of Zhou et al. (the first two columns of the rotation matrix), in the OpenCV convention with z along the fingers and x to the right. Grasp states are absolute, not differences: a gripper open-close value, or fingertip positions in each wrist frame. Report §2.1.3, Fig. 3

Embodiment	Ego pose	Effector poses	Grasp state	Width	Example source
Camera motion	9	none	none	9	ViPE + Depth Anything 3 poses
Autonomous vehicle	9	none	none	9	Hyperion logs, front-wide frame
Single-arm robot	none	9	1	10	DROID, Bridge, Fractal, UR
Dual-arm robot	none	2 × 9	2 × 1	20	RoboMIND dual Franka
Humanoid	9	2 × 9	2 × 1	29	AgiBot World
Egocentric human	9 (head)	2 × 9 (wrists)	2 × 15 (fingertips)	57	proprietary head-mounted capture

Report Fig. 3, §3.2.3; widths confirmed by the model card and the Diffusers embodiment table, which also lists Galbot (30) and Push-T (2). The card lists UMI as 9D and the pipeline table as 10D.

Each embodiment domain k gets its own input and output projection, z = W_in^(k)x + b_in^(k) and x̂ = W_out^(k)z + b_out^(k) (Eq. 1, 2), trained from scratch with the shared backbone. Channels are normalised per dimension to roughly [−1, 1]. Report §2.1.3, §3.2.3 The released code zero-pads every action to 64 channels and reserves 32 domain slots. transformer/config.json

Why 6D rotations

Zhou et al. showed that every rotation representation with four or fewer numbers is discontinuous as a function of SO(3). Quaternions double-cover it (q and −q are the same rotation), Euler angles wrap and lock, axis-angle flips at π. A regression target with a discontinuity forces the network to approximate a jump. The 6D representation is continuous and is mapped back to a rotation by an orthogonalisation. For a diffusion model the argument is sharper: the learned velocity is a conditional mean, and a target that can appear as q or −q averages toward zero. Zhou et al., CVPR 2019; the diffusion point is Inference

The report says only that the predicted 6D is converted "using SVD". A standard construction stacks the two predicted columns with their cross product and projects to the nearest rotation, R = U·diag(1, 1, det(UV^T))·V^T. Which construction the code uses is not stated. Inference

Three generation modes, one backbone

With latent video frames v_1:N and action tokens where a_t is the transition from v_t−1 to v_t, the three modes differ only in which tokens are noised: Report §2.2.2, Fig. 4

Mode	Distribution	Clean (conditioning)	Noisy (denoised, in the loss)
Forward dynamics	p(v_P+1:N \| v_1:P, a_P+1:N, l)	context frames, all actions	future frames
Inverse dynamics	p(a_2:N \| v_1:N, l)	every frame	actions
Policy (joint)	p(v_P+1:N, a_P+1:N \| v_1:P, l)	context frames	future frames and actions

Because σ is sampled per modality during training, the network also sees mixed states such as clean actions with half-noised video, the same trick that lets one network serve several conditionals in UniDiffuser-style models. Whether video and action share one schedule at inference in policy mode is not disclosed. Inference

One ablation tests whether the modes help one another, on Push-T with the Edge model: three single-mode runs of 2K steps against one joint run of 6K steps. The joint model cuts inverse-dynamics MSE by 72% (1.11×10⁻³ to 3.09×10⁻⁴) and lifts policy coverage from 74.1% to 77.3%, at a cost of 0.9 dB forward-dynamics PSNR. One dataset, one seed. Report App. E.4, Tab. 31

Pseudo-actions and the noise they carry

Every pose stream is estimated, not commanded. Egocentric data (67.4% of the 61.3K action hours) uses head-camera pose deltas, wrist deltas from 21-keypoint hand tracking and fingertip positions. Camera motion (7.5%) uses poses estimated with ViPE and Depth Anything 3 on web video, with jittery clips filtered out. Driving (16.3%) is transformed into the front-wide camera frame. Robotics (8.7%, 5.36K hours across AgiBot, Franka, Google Robot, WidowX, UMI and UR) uses end-effector flange deltas. Report §2.1.3, §3.2.3, Fig. 9, Tab. 4

A pose delta records what the body achieved, not what the controller asked for. Tracking error, compliance and contact forces are folded into the label, so the forward model learns something closer to state transitions than to control, and a policy that outputs deltas still needs a controller to realise them.
Estimator error from ViPE, Depth Anything 3 and hand tracking enters the targets directly; the report filters the worst clips but never quantifies the residual noise.
A human wrist is not a robot flange: reach, speed limits and the grasp-state semantics (fingertips against a one-number gripper) differ, and per-dimension normalisation erases absolute scale across domains.
The prior is dominated by humans, vehicles and cameras. Robots contribute under a tenth of the hours, and the single-arm Franka subsets used in the synergy study hold 23 and 4 hours.

All four points: Inference from Report §3.2.3 and Fig. 28.

Chunk length, frequency and horizon

Disclosed: base-model actions at 10 to 30 Hz over 16 to 400 frames for inverse dynamics and policy; DROID forward dynamics with 16-step end-effector chunks predicting 16 frames; driving inverse dynamics on 6-second clips at 10 FPS; the DROID policy with 32 absolute joint positions at 15 Hz, a 2.13-second chunk that the Franky controller executes in full. Report §6.3.1, §6.2.5, §4.2.5

Not disclosed: the policy's inference latency per chunk, whether any of the 32 steps overlap with the next prediction, temporal ensembling, how proprioception is encoded, and the action dimension of the DROID head. Report, silent

The complete input-to-output flow

 inputs                     encoders                        one packed sequence                    MoT backbone (L layers)            outputs
 ──────                     ────────                        ───────────────────                    ───────────────────────            ───────
 text prompt / JSON  ─────▶ Qwen3-VL tokenizer ──┐
 image or video      ─────▶ ViT 16px + 2×2 merge ├──▶  [ AR: l₁ … lₙ  (ViT…)  EOS  BOG ]  ──▶  reasoner tower: causal attn  ──▶  next-token text
    (understanding)          DeepStack           ┘              │  keys/values cached                 over AR only
                                                                  ▼  temporal gap 15,000
 image / video frames ────▶ Wan2.2 VAE (frozen) ──┐   [ DM clean:  v₁…v_P   s…   a… ]      ──▶  generator tower: full attn      ──▶  VAE decode → video
 control video        ────▶ same VAE            ──┤   [ DM noisy:  ṽ…      s̃…   ã… ]           over [AR ; DM]                   audio VAE → 48 kHz
 audio                ────▶ audio VAE (frozen)  ──┤                                            flow matching, 4 to 50 steps       W_out^(k) → actions
 actions (9 to 57 D)  ────▶ W_in^(k), per domain ─┘                                            CFG: text (and control) weights

Report §2.1 to §2.4, §5.3.1, Tab. 21. Redrawn from Fig. 5; sequence order is modality-blocked, time alignment comes from the RoPE coordinate.

Five paths through the same weights

(a) Understanding. Text and ViT tokens form the whole sequence; no DM tokens exist and the generator parameters are never touched. The model is a VLM decoding text. Report §2.2.2

(b) Text or image to video, with optional audio. The sequence is [S_AR, ṽ_1:N, s̃] for text-to-video, and [S_AR, v_1:P, ṽ_P+1:N] when P conditioning latent frames are given (P = 1 is image-to-video, P = 2 or five pixel frames is video-to-video). The reasoner runs once, its keys and values are cached, and the generator denoises for 50 steps with guidance 6 and shift 10. Report Eq. 3 to 5, §5.3.1, Tab. 21

(c) Control-conditioned transfer. Edge, blur, depth, segmentation or world-scenario-map video is encoded by the same frozen VAE and placed as clean DM tokens, [S_AR, v^ctrl_1:N, ṽ_1:N]. There is no ControlNet branch. Guidance uses two weights and three evaluations per step: full conditioning, prompt only, and control with a negative caption; defaults are text 3 and control 1.5. Report Eq. 6, §6.2.4, Tab. 21

v̂ = v_full + (w_ctrl − 1)(v_full − v_prompt) + (w_text − 1)(v_full − v_ctrl,neg) one consistent reading of the report's description; the exact combination is not written out

(d) Observation and actions to future video. Context frames and the action sequence are clean, future frames are noisy; forward dynamics samples with 50 steps, guidance 1 (no CFG) and shift 5. Report Fig. 4, Tab. 21

(e) Observation and goal to video and actions together. In policy mode future frames and actions are both noisy. The DROID policy takes a 540×640 canvas (wrist view above two external views) plus proprioception and the official short task instruction, runs 4 denoising steps with guidance 3 split across two GPUs, skips decoding the video latents, and emits 32 joint positions. Report §4.2.5, Tab. 21

Where reasoning enters the generator, and what "think before it acts" means

In every generation mode the reasoner's contribution is its cached keys and values, read by the generator through Eq. 8. The AR subsequence is the prompt, [l₁…l_n, EOS, BOG]; the model does not decode a chain of thought before denoising. The explicit version exists as a separate call: prompt upsampling, where the reasoner, or an external LLM, expands a short request into a typed JSON scene program that becomes the prompt. Report §2.2.2, §6.3.2

Demonstrated. Initialising the understanding tower from the Cosmos 3 reasoner instead of Qwen3-VL raises the PAIBench text-to-video domain score from 73.7 to 75.7, with the largest gain on robotics (+4.8), in one run per arm at reduced scale. App. E.1, Tab. 28
Demonstrated, with a caveat. Structured prompts improve generation, but every generation benchmark rewrote prompts with Claude Opus 4.6 or 4.7, and the arena entry used GPT-5.5. The Cosmos reasoner is not the upsampler in the headline numbers. §6.2.1, §6.2.2, App. B.7
Not demonstrated. No experiment shows reasoning improving an action. The DROID policy is prompted with the short official instruction and no reasoning step. §4.2.5
Marketing framing. "A reasoning block to first interpret what is happening in a scene, then ... a generation block" describes the causal mask and the upsampler, not a planning loop. NVIDIA blog
A structural limit. Because AR rows never attend to DM columns, the reasoner cannot revise its reading of the scene after seeing an imagined future within one pass; imagine-then-reason needs a second call with the generated video fed back through the ViT. Inference from Eq. 7

Is there any explicit 3D?

Geometry is implicit. There is no depth or point-map output head; depth appears only as a transfer control input. Camera pose enters as an action stream (9D relative poses), which is a form of camera conditioning through the action channel rather than a pose-token interface. The reasoner can emit camera-frame 3D boxes as JSON text, supervised from MADS and scanned scenes. Multi-view data is tiled into a single canvas with the layout described in the prompt; MADS has seven synchronised cameras with per-camera scenario maps, but joint seven-view generation is never evaluated and the driving transfer benchmark is single-view. Report §2.1.3, §3.1.2, §3.2.1, App. B.5, §6.2.4

The model card states the consequence plainly: "Because the model lacks an explicit physics simulator, 3D geometry, 4D space-time evolution, object permanence, contact dynamics, and physical laws are only approximated." Cosmos3-Nano-Policy-DROID card, Limitations

Data, curriculum and an evaluation audit

What each tower was trained on

Stage	Reasoner	Generator
Pre-training	22.0M samples: OCR 42.9%, 2D grounding 16.5%, visual QA 11.3%; dedup, Gemma-4-31B judge at threshold 2	767M images and 347.7M clips from 7.8B and 3B raw; 138.9M clips with audio
Mid-training	none	15.6M images, 74.7M clips (46% filtered web, 43.9% embodied domains, 10.1% hard cases), 18.8M audio clips, 8.4M action episodes (61.3K h), 3M transfer videos plus 1.1M MADS samples; mix 10% image, 32% video, 8% video with audio, 25% action, 20% general transfer, 5% driving transfer
Fine-tuning / post-training	2.2M samples, video 50%, judge threshold 5; AV action CoT (10K human, 1.1M auto), robot action CoT, surgery VQA, smart-infrastructure sets	Text2Image: 20K + 2K steps; Image2Video: 10K iterations at 480p, 189 frames, about 50B tokens; Policy: DROID at 360×640

Report §3.1, §3.2, Tab. 3, Tab. 6, Fig. 8, Fig. 9, §4.2.3 to §4.2.5.

Five synthetic sets were built in simulation and released: SDG-PhyxSim (rigid, articulated, deformable, fluid and optical physics), SDG-RobotSim (six to eight embodiments), SDG-DriveSim, SDG-SynHuman and SDG-Warehouse. Their ablation fine-tunes Nano on each and scores PAIBench-G text-to-video: every change is within ±1.5 points, from one run each with no seeds, and the Human domain drops under every synthetic source, including the synthetic-human one. Report §3.2.1, App. C.7, Tab. 26

The audit

The board below reads every benchmark the report leans on through the same checklist: what it measures, which axis of a world model or policy that is, whether Cosmos and the baselines ran the same protocol, how many seeds or raters stand behind it, whether any interval is reported, and whether the model trained on data from the evaluation's source. Filter it by axis or by flag. Inference, from the Report tables cited in each row

What the evidence supports, axis by axis

Policies. The RoboLab gain is large and clean on its own terms: 39.7% against 28.1% for π0.5 on 120 tasks with ten rollouts each, a gap of about six binomial standard errors if the 1,200 episodes per model were independent, and fewer once they are clustered by task. It targets instruction phrasing (vague, default, specific) and task difficulty, on the same Franka-DROID embodiment the policy trained on, in simulation after training on real data. RoboArena covers unconstrained real objects and scenes on that same embodiment, from an undisclosed number of pairwise votes at one timestamp. MolmoSpaces probes controlled object, colour and camera variation. LIBERO-10 is the only new embodiment, reached by fine-tuning to 97.4% in-distribution success, not zero-shot. No result tests embodiment transfer without training. Tab. 19, Tab. 20, Fig. 26, Fig. 27 axis reading: Inference

World models. Visual quality is measured well (PAIBench-G quality, DOVER, the HUE visual-integrity axis). Physical plausibility is measured by judges and by Physics-IQ, whose best Cosmos score of 63.4 is against a real-versus-real ceiling of 100. Action controllability has three pieces of evidence: camera-trajectory error re-estimated from generated video, driving inverse dynamics, and one video-action consistency check in which the policy's predicted video matches a simulator replay of its predicted actions at 23.19 dB (external camera) and 17.33 dB (wrist). Downstream utility, meaning closed-loop success of a policy trained on Cosmos-generated data or ranked by Cosmos as a simulator, is not measured anywhere in the report. Tab. 12 to 18, App. E.5 Inference

Three numbers that do not agree with each other. The RoboLab text says DreamZero reaches 25.2% with specific instructions; Table 19 says 23.9%. Cosmos HUE says each video receives "up to 16" questions in one paragraph and "up to 20" in another. The report's serving figure labels 286.33 s as text-to-video on an H100 NVL, while the repository's table gives 311.13 s for text-to-video and 286.33 s for image-to-video. §6.2.5, App. F inference_benchmarks.md

Robotics use and the limits of the closed loop

Three roles, three levels of evidence

Synthetic data engine. Supported for visual quality and for control adherence in transfer; not supported for the property that matters, whether a policy trained on the output performs better. No such experiment is reported. Inference from §6.2, App. C.7

Learned simulator for policy evaluation. The report presents robotics forward dynamics as a route to "policy evaluations with video models", and backs it with 16-frame DROID PSNR (26.04 dB for Super against 22.99 for Ctrl-World) and the single consistency check above. The test that would establish it, rank correlation between success judged inside Cosmos and success on a robot across many policies, is absent. §6.2.5, Tab. 18 Inference

Policy backbone. The strongest evidence in the report: first place on RoboLab, RoboArena and MolmoSpaces at submission time, with the same checkpoint and no benchmark-specific tuning, and mid-training that makes LIBERO-10 adaptation faster (24.6% against 0.0% at 500 iterations). §6.2.5, Tab. 19, Tab. 20

What stands between it and closed-loop training

Latency. Measured, single B200, BF16, batch 1: Nano text-to-video takes 4.56 s at 256p, 33.2 s at 480p and 114.9 s at 720p for 189 frames (7.9 s of video), so only 256p runs faster than real time. Super takes 114.4 s at 480p and 407.5 s at 720p. The 4-step distilled Super checkpoints (DMD2, no CFG) claim 17.5× to 25× fewer network evaluations, a derived estimate the card labels as not measured. The policy's own latency is not published. inference_benchmarks.md, 4Step card
Compounding error. The generator is bidirectional over the clip, so it cannot stream with a causal key-value cache; long rollouts must be chained chunk by chunk through video-to-video conditioning on the last two latent frames, and each chunk inherits the previous one's errors. No multi-chunk drift is measured. video_temporal_causal: false Inference
Action-following fidelity. One consistency number at one horizon on one simulator. There is no counterfactual test (same start, different actions, do the videos diverge the way the physics does?), which is the test that separates a controllable simulator from a model following its prior. Inference
Contact and deformables. HUE's physical-laws axis sits at 85.4 for Super against 90.5 for real video, the synthetic physics set moves PAIBench scores by under a point, and the card lists contact dynamics among what is "only approximated". Tab. 32, Tab. 26 Card
Plausible is not causal. Judges and human raters score whether motion looks right. Only Physics-IQ compares against what physically happened, and it is scored on pixels and masks rather than on object states. Inference from §6.2.2
Sim-to-real in reverse. Every synthetic source lowered the Human domain score, which the report reads as a sim-to-real gap in generation. For policies trained on generated video the gap is unmeasured. App. C.7

Where it sits among its neighbours

A world model can be a renderer (produces observations from a state or description), a simulator (advances the state under an action) or a planner (chooses the action). The map below places four systems; hover a point for the reasoning. Taxonomy after World Labs' functional taxonomy; placements are Inference

The three experiments a reviewer should demand

Does the predicted video help the action? Retrain Cosmos3-Nano-Policy-DROID without the auxiliary video loss and with video tokens absent at inference, at matched compute, and score RoboLab 120 × 10 over three seeds. The report trains the policy to predict video, then discards it at inference, and never tests whether the video pays for itself. Cost, estimated: the policy's iteration count is not disclosed. At 16.2M video tokens per GB200-hour and roughly 3K tokens per sample, each billion training tokens costs about 60 GPU-hours, so a run of 20 to 40B tokens is 1,200 to 2,500 GPU-hours and two arms by three seeds come to about 7,000 to 15,000 GB200-hours, plus 7,200 simulated episodes. estimate from Tab. 8 throughput
Is it a valid policy evaluator? Take ten or more DROID policies of varied strength, estimate each one's success inside Cosmos forward dynamics and on real robots or RoboLab, and report rank correlation with a bootstrap interval; add a counterfactual set where the same start frame is rolled out under eight different action chunks. Generation is cheap here (about 10,000 chunk rollouts at a few seconds each, under 100 B200-hours); the cost is on the order of 1,000 real rollouts. estimate
Does generated data train better policies? Generate matched volumes of Cosmos video, label it with Cosmos inverse dynamics, and train a fixed policy on real only, real plus simulator data, and real plus Cosmos data; report real-robot success with confidence intervals. Ten thousand 8-second clips from Super at 480p on B200 cost about 320 GPU-hours to generate, before policy training. estimate from the measured 114.4 s per clip

Concept guide

Each card gives a mechanical picture and, where it helps, one line of mathematics. Click a card for the analogy.

Interactive tour: one mechanism per step

Fourteen steps, each isolating one mechanism. Drag a 3D scene to orbit it, click it and scroll, or use the + and − buttons, to zoom, and use the segment bar under each scene to highlight, isolate, hide or explode a component. Hover a component for its name and tensor shape. The depth switch changes how much each caption says; the scene stays the same. Arrow keys move between steps when the tour has focus.

Step 1 of 14

Speed 1×

Modalities text ViT vision VAE video latents audio action special (EOS, BOG) noise / noisy token

Tokenisation factory

Report §2.1 Card/Code rounding conveyor schematic

Duration 5 s FPS Tier Aspect audio Actions

Every input is cut into pieces the transformer can read. Words become text tokens. For understanding, pictures become ViT patches. For generation, video goes through a frozen compressor that shrinks time by 4 and space by 32. Sound becomes 25 tokens a second. Each robot command becomes one token after a small per-robot projection. Move the sliders and watch the count.

Two vision encoders serve two towers: a trained ViT (16×16 patches merged 2×2, DeepStack features) for understanding, and the frozen Wan2.2-TI2V-5B VAE for generation. The VAE is causal, so clips are 4k + 1 frames. At 720p the 45 latent rows are odd, and the released pipeline rounds the patch grid up to 23. Audio is a frozen 48 kHz stereo VAE with hop 1,920. Actions pass a per-embodiment linear map into d_model.

Only the ViT is trained with the backbone; both VAEs are frozen, so the generator's ceiling on fidelity is the Wan2.2 decoder's. A VAE token carries 48 channels × 2 × 2 = 192 numbers before the linear projection to 4,096 (Nano) or 5,120 (Super). The 74,000-token packed context is why 720p stops at 300 frames: 75 latent frames × 920 = 69,000 tokens. One action token per pixel frame is the pipeline's convention; placement in time comes from the RoPE step, not from alignment in the sequence.

A post office that repacks parcels of every shape into standard crates before they share one conveyor.

Governing expression

video tokens = latent frames × tokens per frame

T′ = (T − 1)/4 + 1 · tokens = T′ · ⌈H/32⌉ · W/32 · audio = ⌈25 · T/fps⌉ · z = W_in^(k) x + b

(B, T, H, W, 3) → VAE (B, T′, H/16, W/16, 48) → patch (B, T′·⌈H/32⌉·W/32, 192) → linear (B, N, 4096)

Token arrangement and the clock on every token

Report §2.2, §2.4, Eq. 3 to 6, 9 token counts schematic

Task

Everything goes into one line. First the words, read left to right. Then the generation tokens: clean conditioning first, noisy targets after, each group ordered vision, sound, action. The line order is not the time order: every token also carries a clock value saying when in the clip it belongs. Switch to "Physical time" to line the tokens up by that clock.

Positions are 3D RoPE coordinates (t, h, w). Words use t = h = w. Video latents vary on all three, with h and w restarting in each vision segment. Audio and action carry time only. The generation block starts 15,000 temporal units after the last word, a gap added because adjacent positions produced saturated, checkerboarded first frames. The time step per token is δt = 6 / TPS, so one physical second is six units in every modality.

TPS is fps/4 for video (the VAE's temporal factor), 48,000/1,920 = 25 for audio, and the sampling rate for actions; TPS_base = 24/4 = 6. The rotary split is 24 temporal, 20 height and 20 width frequency pairs per 128-wide head, interleaved, with base 5×10⁶. Duration and FPS also appear as text in the JSON prompt; the ablation (App. E.2) found RoPE modulation plus text control best, composite 9.81 against 8.51 with neither.

Luggage on one belt, grouped by type, each bag tagged with its flight time.

Governing expression

one line of tokens, one clock value per token

S_policy = [S_AR, v_1:P, ṽ_P+1:N, ã_P+1:N] δt = TPS_base/TPS, TPS_base = 6

t₀ = t_{last AR} + 15000 · t(latent frame k) = t₀ + 24k/fps · t(audio j) = t₀ + 0.24 j · t(action i) = t₀ + 6i/f_a

One Mixture-of-Transformers layer

Report §2.3, Fig. 5 shapes from config slab sizes schematic

Model

One layer holds two complete sets of machinery side by side. A word token only ever uses the left set, a video, sound or action token only the right. They meet in one place, the attention step, where the right side may read the left and the left may never read the right. Pick a token to follow its path.

Each tower has its own layer norms, query, key and value projections, output projection and feed-forward network, all copied from Qwen3-VL at the start. Routing is fixed by position (AR prefix left, DM suffix right), so there is no learned router. Attention is computed once over the concatenated keys and values with the mask of step 6. The released config uses QK-norm in both towers and grouped-query attention with 8 key-value heads.

Per layer and tower, Nano holds Q: 4096×4096, K and V: 4096×1024 each (8 KV heads × 128), O: 4096×4096 and a SiLU-gated FFN of 3 × 4096 × 12,288, about 193M parameters, times 36 layers and two towers, which is where the two 8B halves come from. The output projection being per-tower is Inference from "attention projection matrices"; the report does not list O separately. The reasoner's K, V for the prompt are computed once and cached across all denoising steps.

Two kitchens sharing one pass-through window: the pastry cooks can read the main kitchen's tickets, the main kitchen never sees the pastry.

Governing expression

left tower: look back only · right tower: look everywhere

h ← h + W_O^τ·Attn(LN^τ(h)W_Q^τ, [K_AR;K_DM], [V_AR;V_DM]) ; h ← h + FFN^τ(LN^τ(h)), τ ∈ {AR, DM}

Q: (B, n, 32, 128) · K, V: (B, n, 8, 128) · FFN: 4096 → 12,288 → 4096 · 36 layers

Flow matching, from noise to data

Report §4.2, Tab. 21 sampler: UniPC toy data, exact field

Steps 4 CFG 1.0 Shift 1.0 Condition

Generation starts from random noise and walks it, step by step, along a direction field toward realistic data. Training only ever shows straight lines from a real sample to a noise sample; averaging many such lines gives curved paths. Take too few steps and the walk cuts corners and lands between realistic outcomes, which on an image looks like blur. Try 1, 4 and 50 steps.

The toy data is six small clusters in two classes, so the ideal velocity field has a closed form and the paths shown are exact, not learned. The faint curves integrate that field with 400 steps; the bright polylines use your step count. Shift reshapes the schedule toward high noise, which Cosmos raises with resolution (up to 10 at 720p). Guidance extrapolates from the unconditional to the class-conditional field.

For clusters N(μ_k, τ²I) the optimal field is v(x,σ) = E[ε − x₀ | x_σ = x], a posterior-weighted mixture of per-cluster affine maps; one Euler step from σ = 1 lands on the posterior mean of x₀, the average of all modes. Cosmos samples forward and inverse dynamics with 50 steps, generation with 50 steps and guidance 6, and the policy with 4 steps and guidance 3. The production sampler is second-order UniPC, which bends less than Euler at equal steps.

Water finding its way downhill from a scattered spray to a few basins: with a coarse map you step over the ridges and land on the slopes between them.

Governing expression

noise → data by following arrows; training arrow = noise − data

x_σ = σε + (1−σ)x₀ · v* = ε − x₀ · x ← x + (σ_i+1 − σ_i)·v_θ(x, σ_i) · σ = s·u/(1 + (s−1)·u)

v_cfg = v_∅ + w(v_c − v_∅) · L = E‖M ⊙ (v_θ(x_σ,σ,c) − (ε − x₀))‖², σ per modality, λ_action = 10

Unified action representation

Report §2.1.3, Fig. 3, Eq. 1, 2 pad 64, 32 domains SVD construction

Embodiment

Every body is described with the same few parts: how a frame moved (3 numbers), how it turned (6 numbers), and how closed the hand is. A car is one moving frame. A robot arm is one hand plus a gripper. A person is a head, two wrists and ten fingertips. A small per-body projection turns any of these vectors into one token the shared network understands.

Poses are relative, ΔT = T_t−1⁻¹T_t, so the model never needs a robot's controller details. Rotation uses the first two columns of the rotation matrix (6 numbers), which is continuous where quaternions and Euler angles are not. The network's 6D output is mapped back to an exact rotation by an SVD; press the button to watch a noisy prediction snap to the nearest rotation.

Per domain k, z = W_in^(k)x + b_in^(k) with W_in^(k) ∈ ℝ^d×d(k), and the inverse map on the way out. The released code pads x to 64 channels and has 32 domain slots. Grasp states are absolute, not deltas. The DROID policy replaces this head with a fresh encoder and an MLP decoder over 32 joint positions, so its numbers do not exercise the unified interface. The projection shown uses polar iteration, which equals the SVD projection UV^T when the determinant is positive.

One shipping manifest for every vehicle: the columns are the same, a bicycle just leaves most of them empty.

Governing expression

move (3) + turn (6) + grip (1 or 15) per part

ΔT_t = T_t−1⁻¹T_t · r = [R_:,1; R_:,2] ∈ ℝ⁶ · R̂ = U·diag(1,1,det UV^T)·V^T

x ∈ ℝ¹⁰ → pad ℝ⁶⁴ → W_in^(k) → z ∈ ℝ⁴⁰⁹⁶ · x̂ = W_out^(k)z + b_out^(k)

Reasoning to generation

Report §2.3.2, §5.3.1, §6.3.2, App. E.1 "think before it acts"

prompt upsampling (explicit plan) cache reasoner K, V denoising step 1 / 50

The reasoner reads the prompt; the generator, at every denoising step, looks at what the reasoner computed. That look is one-way. Turn on prompt upsampling to see the explicit version: the reasoner first writes out a detailed plan as text, and that plan becomes the prompt the generator reads.

Demonstrated: initialising the understanding tower from the Cosmos reasoner lifts PAIBench text-to-video domain score 73.7 → 75.7 (robotics +4.8), one run per arm; reasoner outputs are cached once and reused for all steps. Claimed, not shown: that reasoning improves actions. The DROID policy is prompted with the short official instruction, and the headline generation benchmarks used Claude Opus or GPT-5.5 as the upsampler, not the Cosmos reasoner.

The upsampler emits a typed JSON scene program (entities, layout, temporal events, camera, audio, controls) through a four-block prompt contract. Since AR rows cannot attend to DM columns, a reasoning trace cannot be revised by the imagined future inside one pass; "imagine, then reason again" requires feeding generated frames back through the ViT in a second call. The blog's "think before it acts" is this causal ordering plus the optional upsampler.

An architect's brief pinned above the drafting table: the draftsman keeps glancing up at it, and the brief never changes while the drawing is made.

Governing expression

plan (optional) → prompt → cached → read at every step

K_AR, V_AR computed once; O_DM^(s) = Attn(Q_DM^(s), [K_AR; K_DM^(s)], [V_AR; V_DM^(s)]), s = 1…S

upsampled prompt l* ~ p_reasoner(· | l, template, image) → S_AR = [l*, EOS, BOG]

Closed-loop rollout and compounding error

Report §4.2.5 latency table error model schematic

Chunk 32 Model error 0.35 World-model tier

The policy predicts a chunk of actions and a short video of what should happen, the robot executes the chunk, and the loop repeats. As a policy, each new chunk starts from a fresh look at the real world, so errors reset. As an imagination engine, each chunk starts from the previous imagined frame, so small errors pile up. The band shows how far prediction and reality drift apart.

The DROID policy executes 32 joint positions at 15 Hz, 2.13 s per chunk, after 4 denoising steps on two RTX Pro 6000 cards; its latency is not published. As a world simulator the costs are measured: on one B200, Nano needs 4.56 s for 7.9 s of 256p video, 33.2 s at 480p and 114.9 s at 720p, so only 256p is faster than real time. The error model is schematic.

The generator is bidirectional over a clip (video_temporal_causal: false), so a rollout longer than one clip is chained by video-to-video conditioning on the last two latent frames, with no key-value reuse across chunks. Drift over chained chunks is not measured in the report. The schematic uses e_k+1 = (1 + λ)e_k + ε for open loop and e_k+1 = ε for a re-observed closed loop, with ε set by the slider.

Dead reckoning against a GPS fix: estimate from your own last guess and the error grows; check the real position every leg and it stays bounded.

Governing expression

open loop: errors add up · closed loop: errors reset each chunk

chunk time = H / 15 Hz · real-time factor = generation time / video time

e_k+1^open = (1+λ)e_k + ε · e_k+1^closed = ε + δ_obs (schematic)

Story mode: a mug on a shelf

One continuous run through the tour with a single example. The robot sees its workspace and is told to put the mug on the shelf. Each stop pauses on the step that explains it; open that step in detail, then come back to the story.

Play the story

What is data-faithful and what is schematic

Data-faithful to the report or code: the token calculator and every shape readout (steps 2, 3, 5, 8), the attention mask structure (6), the mode layouts and position rules (4, 9), the flow-matching objective, shift and exact toy field (7), the action widths and projection shapes (8), the latency and chunk figures (12), the audit numbers (13). Schematic: robot and mug scenes (1, 12), cell counts in the VAE volume and token ribbons (3, 4, 9), slab sizes in the layer model (5), the transfer renderings and their drift (11), the error-growth model (12) and the taxonomy placements (14).

Checked on 22 Sep 2026 in headless Chromium with software WebGL, at 1440×900, in dark mode, with reduced motion and at 390×844: all fifteen panels render; every button, slider, select, checkbox, segment control and camera button on every step was exercised; story mode ran all twelve stops; no page errors and no horizontal scroll. Tooltips and orbit were checked with a mouse pointer only; touch orbit was not tested on a physical phone.

Corrections before the answers

Strategic position and problem formulation

What NVIDIA means by a world foundation model and by physical AI

Why each standalone model class falls short

Targets, and where the open weights fit the business

Architecture and mechanism

The Mixture-of-Transformers layer

How the two objectives are trained

Tokenisation, modality by modality

Multimodal position embedding

Nano against Super

Action as a first-class modality

One interface for six embodiments

Why 6D rotations

Three generation modes, one backbone

Pseudo-actions and the noise they carry

Chunk length, frequency and horizon

The complete input-to-output flow

Five paths through the same weights

Where reasoning enters the generator, and what "think before it acts" means

Is there any explicit 3D?

Data, curriculum and an evaluation audit

What each tower was trained on

The audit

What the evidence supports, axis by axis

Robotics use and the limits of the closed loop

Three roles, three levels of evidence

What stands between it and closed-loop training

Where it sits among its neighbours

The three experiments a reviewer should demand

Concept guide

Interactive tour: one mechanism per step

The problem: three specialists, three failures

Governing expression

Tokenisation factory

Governing expression

Video VAE compression

Governing expression

Token arrangement and the clock on every token

Governing expression

One Mixture-of-Transformers layer

Governing expression

Attention mask explorer

Governing expression

Flow matching, from noise to data

Governing expression

Unified action representation

Governing expression

Three action modes, one backbone

Governing expression

Reasoning to generation

Governing expression

Video transfer and control strength

Governing expression

Closed-loop rollout and compounding error

Governing expression

Evaluation audit board

Renderer, simulator, planner

The whole path, and the three limits that matter most

Story mode: a mug on a shelf

What is data-faithful and what is schematic