Research Ideas

The talk argues that once a frontier model is capable enough, dexterity becomes a reward problem, noisy data a representation problem, sim-to-real an optimization an agent can run, and low-level control a question of output format. This lecture walks the twelve hot takes with an evidence grade on each, the full 43-row RoboDojo board, the ENPIRE loop, VIRAL's four sim-to-real gaps and RoboICL's context grammar, and adds six labs that run in the page: reading pass@k, fitting a simulator, hardware as code, action tokenization, control-rate budgets and structure against scale.

• Astra leads RoboDojo at 22.48% average success and 31.00% on open tasks, yet scores 4.00% on precision, where the best public policy reaches 20.42%; GPT-5.5 manages 0.88% through the same interface.
• Format beats content in context: naive one-shot demonstrations cut Astra's success from 22.9% to 17.9% and 12.9%, while RoboICL's observation, action, receipt, observation grammar adds 20 to 27 progress points.
• ENPIRE's 99% is pass@8 with in-context retries; three 60% subtasks with full recovery give 99.8% at k = 8 and 21.6% on the first try.
• Fitting a simulator over a multi-step horizon identifies parameters that one-step fitting cannot, and the error that remains concentrates at contact, the gap the talk calls the last one.

Twenty-three levels take MuJoCo from what a simulator computes, through MJCF, the Python and JavaScript bindings and rigid-body mechanics, to controllers, contact, Gymnasium, reinforcement and imitation learning, vision, language conditioning, world models, domain randomization, system identification and sim-to-real. Every lab is MuJoCo itself, compiled to WebAssembly and running in the page, and every measured number in the text comes from a script in the companion repository.

• Claims carry their kind: established MuJoCo behaviour, derivation, implementation detail, engineering recommendation or research-level interpretation, with anything unconfirmed marked unverified.
• Measured, not paraphrased: a resting ball sinks 0.37 mm into the floor at 2 kg and at 0.1 kg alike, because MuJoCo specifies soft contacts as a reference acceleration rather than a force.
• A pytest suite checks every lesson script, model, broken-model case and project solution against MuJoCo 3.14.0, and a content checker flags any lesson whose embedded code differs from its source file.
• The thirteen-case debugging clinic walks symptom, diagnosis, root cause, fix and lesson for common faults such as degrees read as radians, xyzw quaternions and stale observations after reset.

Mattia Piccinini and Gastone Pietro Rosati Papini's July 2026 course sorts every way of adding physics to a learned model by where the prior enters: the inputs (physics-guided), the architecture (physics-encoded) or the loss (physics-informed). This guide walks the three routes with their equations, from PINNs, Lagrangian and Hamiltonian networks and Neural ODEs to the lecturers' model-structured networks and the nnodely library, reads the racing case studies against their own tables, and adds four labs that run in the page and a timestamp for every topic in the 8 h 38 min of video.

• The boundary rule that makes the taxonomy usable: preprocessing must be frozen or pre-trained to count as physics-guided; once it trains with the model, the method is physics-encoded.
• A neural compositional module is a scheduled Hammerstein-FIR local-model network: a static input map, partition-of-unity regime weights, an FIR layer whose trained weights read as an impulse response and expose dead time, and a residual.
• Window length comes from the residual spectrum: content below 3 Hz gave 1.25 s of history at 20 Hz (n = 25), and residual power below the noise floor flags overfitting.
• A controller trained by backpropagating a tracking loss through a frozen learned simulator needs no new data and inherits every simulator error; in the page's fourth lab a simulator 40% too light turns 3.3% overshoot into 13.7% on the real plant.

Cosmos 3 folds NVIDIA's Predict, Transfer and Reason model lines into one set of weights and makes action a first-class modality. This teardown walks the mechanism token by token: two parameter towers that meet only in attention, a causal-and-full attention mask, a frozen video VAE at 4×32×32, a 3D RoPE whose clock puts video, audio and action on one physical time axis, rectified flow matching with a noise level per modality, and one action interface for six embodiments. It ends in a fourteen-step 3D tour and a story-mode run.

• Reasoner and generator are trained in stages, not jointly, and the generator reads the reasoner only through cached keys and values; "think before it acts" is the causal mask plus an optional prompt upsampler.
• A 5-second 720p clip is 31 latent frames of 23×40 tokens, 28,520 in all, because the released pipeline rounds the odd 45-row latent height up; the 74,000-token context caps 720p at 300 frames.
• The #1 open-weight text-to-image arena entry was a loop with GPT-5.5 rewriting prompts and Gemini 3.1 Pro judging, not the checkpoint alone.
• The DROID policy denoises future video alongside every 32-step action chunk and then discards it; the report never trains the policy without it.

V-JEPA 2 learns a video encoder by predicting masked tubes of video in a feature space it invents; V-JEPA 2-AC freezes that encoder, trains a 300M block-causal predictor on 23k DROID trajectories and plans with the cross-entropy method on Franka arms in two unseen labs; H-JEPA (arXiv 2610.06805, Oct 2026) stacks action-conditioned JEPAs at several timescales, each in its own latent space, and plans top-down. The lecture takes each model apart in sixteen parts: tokenization and 3D-RoPE, the masking and EMA recipe and why it does not collapse, the block-causal predictor, the CEM planner, SIGReg, slow-feature collapse, hierarchical planning and how to evaluate a world model. Sixteen labs run in the page, among them a 12-D latent world model you fit and plan with, a linear JEPA trained live, the attention mask, the released mask sampler and the Fréchet fidelity metric. The earlier teardown, with its 3D tour and evidence audit, stays at /notes/v-jepa/teardown/.

• DROID's video files are timestamped at 60 fps for 15 Hz captures, so the released V-JEPA 2-AC loader samples one frame per second of real motion, not four: measured on two raw episodes and stated independently by H-JEPA.
• On diverse scenes a SIGReg-only JEPA collapses to the scene identity; H-JEPA proves no regulariser on the marginal law can prevent it and shows an inverse-dynamics term is the barrier that works on DROID.
• Changing only the latent space in which goal distance is measured moves success by up to 4× (FourRoom, 9.3% to 38.7%), and subgoal-tracking costs fall monotonically along expert paths where full-horizon costs do not.
• Running the released sampler: V-JEPA 2's masks hide 62% and 81% of tokens per clip and the encoder sees 25% and 6%, not the "about 90%" usually quoted; every deployed V-JEPA 2-AC plan uses horizon 1 and 16 s per action.
• In the lecture's toy latent world model, CEM with the released planner's momentum reaches 20 of 20 random goals with the camera as trained, 16 of 20 at 60° of rotation and 2 of 20 at 90°: a fitted predictor and a frozen encoder plan well only inside the viewpoint they were fitted on.

World Labs released Atlas as a blog post with no paper, no code and no evaluation artifacts. This teardown walks the whole machine anyway, from what a world model is asked to do, through a multimodal autoregressive diffusion transformer and its rectified-flow sampler, to the splat a robot is driven through, and marks every claim as disclosed, inferred or speculative.

• The preference study gave Atlas native camera paths and every baseline a text description of the path, which World Labs acknowledges, so it measures the input format as much as the generator.
• Only one reconstruction baseline is labelled as posed, and against that one the margin is 25.3 against 28.7 with a tie on Tanks and Temples.
• No consistency metric is reported for the one-minute demo: no revisit PSNR, no loop-closure error. A controlled revisit test would settle it.
• At roughly 14,400 tokens per 1440p frame, a minute of video is about 5.2M tokens, so the scene has to be produced in chunks from a selected context.