Language will eat robotics, slowly, then suddenly.

The argument on one slide

Once general capability in the base model outgrows the robotics-specific stack, four classic bottlenecks stop being research problems and become loops a coding agent can close. Tonghe Zhang, Saturday Robotics #33 Talk

The talk's word for this is that frontier models "overflow absolute intelligence". The write-up reads it as general capability spilling over into robotics, and names GPT-6 Astra as the first model where that has started. Strip the slogan away and the talk makes four reductions and two meta-claims. Click a bottleneck to see what it used to cost and what the talk says replaces it.

Two meta-claims frame the rest

1. End-to-end wins wherever it is feasible. A System 1 and System 2 split is a stage on the way, not the destination. Talk
2. Structure is a temporary scaffold. Every hand-designed interface buys performance at today's compute and costs scalability later. Talk

How to read this lecture

Every claim carries a grade, because the evidence behind the twelve hot takes ranges from controlled benchmarks to a sentence said in discussion.

The switch at the top sets the depth. Overview keeps the claims and the labs. Lecture adds the explanations. Seminar adds protocol detail, derivations and the critique a reviewer would raise.

Seminar mode shows extra protocol notes in most parts.

Twelve hot takes, and how much to believe each

The talk and the discussion after it produced twelve claims. Each card gives the claim in my words, the grade of its best evidence, and a slider for your own credence that it holds by 2030. Open a card for the strongest counter-evidence and the experiment that would settle it. Your answers stay in this browser and come back in Part 15.

What GPT-6 Astra actually showed

Astra ranks first of 43 entries on RoboDojo and is still far from reliable: 22.48% average success, 4.00% on precision tasks. Zhang et al., arXiv 2609.24170, leaderboard snapshot of 10 Sep 2026 Measured

OpenAI released GPT-6 Astra in September 2026. The launch page claims no robot control at all: it shows CAD tool use (95.9% geometric overlap on BenchCAD with tools), a Blender house turned into a walkable Unreal Engine 5 scene, and a schematic routed into a manufacturable PCB. Write-up The robot evidence came from other groups within weeks, and the most careful of it is the RoboDojo early evaluation: three language models run as the policy on all 42 tasks, with Astra and GPT-5.5 at the official 50 episodes per task (2,100 trials each).

In-context learning is not automatic

The same paper tested one-shot demonstrations on 340 matched task-layout pairs, the demonstration coming from a different layout of the same task. Both kinds of demonstration made Astra worse. Measured

RoboICL, from a different group, gets the opposite result by changing the format of the context rather than its content: +20 to +27 progress points over official zero-shot Astra in every category of 30 RoboDojo tasks, and on three real-robot tasks a rise from 14.45 to 63.33 to 78.89 mean progress with zero, one and three demonstrations, all without weight updates. Measured Part 08 takes that format apart.

Where it wins, and where it breaks

Community demos collected in Awesome-Astra-Embodied-AI add the colour: a replayable MuJoCo and Blender simulator built from multi-view robot data and one prompt, simulated hands on a Rubik's Cube, joint-to-end-effector control learned in about 30 minutes, and Astra planning for FluxVLA, SONIC on a Unitree G1 and whole-body controllers. Demo

The measurements support Astra as a policy for semantics and generalization. They do not yet support it for precision, contact or speed. My summary of Part 03 Reviewer

Dexterity is a reward problem

The difficulty runs Reasoning >> Dexterity >> Speed. Language models already hold the hardest one, and a coding agent makes a good reward model for the second. Talk Talk

If a reliable reward exists, dexterity stops being the fundamental obstacle: reinforcement learning, search or plain retrying can climb toward it. The new step is who writes the reward. A coding agent can turn demonstrations, sensor streams or camera views into a binary or scalar verifier, which is what ENPIRE does.

ENPIRE: the loop, concretely

ENPIRE (Xiao, Xie, Zhang et al., arXiv 2606.19980, CoRL 2026) is the speaker's own framework, from NVIDIA, CMU and UC Berkeley. Its name spells the loop. Click a stage, or let it run.

The agents are off the shelf: Codex with GPT-5.5, Claude Code with Opus 4.7, Kimi Code with Kimi K2.6. Tasks include organizing a pin box, fastening and cutting zip ties, pin and GPU insertion, Push-T and tool use. For zip-tie insertion the agents wrote the reward themselves, from object detectors and segmentation models on two cameras. Write-up The paper reports a 99% success rate on these tasks, and faster progress when a team of agents shares a robot fleet. Measured

Verification is the currency

The discussion's larger bet: break a hard-to-verify task into semi-verifiable pieces, then brute-force the self-improvement loop. ENPIRE scores each subtask of a long rollout for exactly this reason. Add sandboxes, tens of thousands of Mac minis for software or a borrowed fleet of 1,000 robots for hardware, and with efficient sampling plus clear verification every robot problem becomes recursive self-improvement. Talk

Why reinforcement learning and not imitation? Because a model sampling a trajectory cannot know in advance that this way of doing it will work. Only an outcome check tells it, and RL is the machinery that learns from the check. Contact-rich loops may still refuse to close on their own and need a carefully designed reward. Talk

Sim-to-real becomes curve fitting

Spend tokens to buy diversity. Bring real scenes and motions into the simulator, fit its parameters until it matches them, and the fitted simulator is an action-conditioned world model. Talk, slide "Automatic real2sim2real" Talk

Before agents: VIRAL's four coverage gaps

VIRAL (He et al., arXiv 2511.15200) is a sim-to-real success. A privileged RL teacher trains on 16 GPUs, a vision student distills from it on 64 GPUs with tiled rendering in Isaac Lab, and the student runs zero-shot on a Unitree G1, succeeding in 54 of 59 consecutive real trials. Measured Its own limitations section still concludes that scaling the recipe to general loco-manipulation "is likely out of reach for the near future", and names four gaps. Click each.

The write-up's reading: most of each gap is labor (tuning materials, authoring scenes, shaping rewards, identifying hardware), and labor is what coding agents automate. Unknown unknowns are the exception, since no authoring finds them before deployment. Seven of VIRAL's fourteen authors are also on ENPIRE, which reads as the same group answering its own limitations section. Write-up

After agents: the simulator is a model you fit

Collect real trajectories, roll the simulator forward from the same starting states under the same actions, and adjust its parameters until the rollouts match:

Two details carry the idea. The rollout horizon H matters: scoring one step at a time hides drift, because a one-step error is smaller than sensor noise. And a simulator that predicts the next state from the current state and action is, by definition, an action-conditioned world model. With a differentiable simulator this is gradient descent; without one the same objective runs under black-box search. Write-up

A fitted simulator is also an evaluator

RoboJEPA (Zholus et al., Meta FAIR, arXiv 2610.10515) supplies the strongest evidence that a learned simulator can stand in for real evaluation. Its imagination error, the error of latent rollouts, follows a second-order power law in compute; downstream planning success improves predictably with compute; and imagination error correlates strongly with real planning success, which the authors propose as a proxy for real-robot evaluation. At 8B parameters it is the largest JEPA predictor trained so far. Measured

The write-up adds the detail: a frozen V-JEPA 2.1-G encoder, 23 public datasets spanning 12 platforms and 26 action spaces, about 15,022 video hours (6,692 with actions), planning by the cross-entropy method toward a goal image, and object-interaction tasks solved only by models trained beyond 10²² FLOPs. Write-up

From craft to agents, and the one gap left

The discussion claims system identification is already solvable by agents: the agent walks a human, or another robot, through the identification protocol and fits the parameters. VIRAL calibrated its cameras and identified its hand by hand. Talk The closest public demo goes partway: Astra ran calibration and system identification from recorded multi-view robot data and produced a replayable MuJoCo and Blender simulator. Demo

The hard part is no longer fitting but the reward that makes an agent pull real data in. The discussion's answer is that offline rewards plus any form of verification are enough to run RL. What remains is contact: tactile models train well, but the instant of contact is still simulated poorly and its error drifts. Solve contact verification, the claim goes, and real2sim is solved. Talk

Hardware is source code

A YAML spec goes in, a parametric and version-controlled robot comes out, and the CAD program is demoted to a viewer. Talk, Implication 1 Talk

The pipeline from the talk has six steps:

1. Spec. A YAML file holds dimensions, masses, actuators and joint limits.
2. Program. The agent writes Python, typically with the open-source parametric library build123d, that builds every part in memory.
3. Export. Each part is written out as a STEP file.
4. Assemble. The program joins the parts, sets joint types, limits and values, and assigns materials and densities. build123d ships rigid, revolute, linear, cylindrical and ball joints, with limits as angular_range and linear_range.
5. Inspect. The agent checks the result; Fusion 360 or similar is opened only to look.
6. Extend. The same agent attaches simulation, system identification and control.

The payoff is hardware that behaves like software: it can be regenerated, branched, diffed, reviewed and optimized in the same loop as its controller. The lab below runs that loop on a toy planar arm.

Why not Onshape, and what already exists

The talk considered Onshape and rejected it for heavy agent use because of API caps. Onshape's published annual limits are 2,500 calls per user on Free and Standard, 5,000 on Professional and 10,000 per full user on Enterprise; past the cap, requests are rejected with HTTP 402. Measured

• OpenAI's Astra launch page: 95.9% geometric overlap on BenchCAD with tools, and a schematic routed into a manufacturable PCB. Write-up
• Onshape Labs' FeatureScript MCP server (early access, August 2026): the model writes FeatureScript, runs it in a document, evaluates the result and fixes errors. Write-up
• A community demo co-designed a quadruped with Astra over 25 loops in five days and trained nine RL motions in simulation. Real hardware is planned, not built. Demo

Generalist, Reward AI, Sunday and Dyna came up alongside the pipeline. All four are robot-learning companies, and the write-up found no public account of agent-generated hardware from any of them, so they are context, not evidence that the pipeline runs in production. Write-up

Noise is a representation problem

The exact details of a motion are redundant. What decides success is the representation the policy learns, and where noise is added back into it. Talk and discussion Talk

Two demonstrations of the same grasp differ in timing, path and wrist angle, and most of that variation does not change the outcome. The discussion's claim is that adding noise back, probably inside representation space rather than raw joint space, determines final success. The target it named is continuous dynamics inside a discrete space: a finite token vocabulary whose embeddings still respect continuity.

• VQ-VAE (van den Oord et al., 2017) encodes an input, snaps it to the nearest entry of a learned codebook, and decodes from that discrete code. It is the natural interface for any modality. Measured
• ViT (Dosovitskiy et al., 2020) is the precedent for brute force: cut an image into 16×16 patches, project them linearly, drop the convolutional prior. Strictly, ViT's patch tokens are continuous embeddings; the discrete step belongs to VQ-VAE. Measured
• VQ-BeT (Lee et al., ICML 2024) tokenizes continuous actions with residual vector quantization. Measured
• FAST (Pertsch et al., Physical Intelligence, January 2025) applies a discrete cosine transform to an action chunk, quantizes the coefficients and compresses them with byte-pair encoding, typically into 30 to 60 tokens. π0-FAST matches the diffusion-based π0 while training up to five times faster, where naive per-dimension binning could not learn the dexterous tasks at all. The price is slower autoregressive decoding. Measured

A language for control

The language model can be the reward model, the trainer, the planner and the controller. To make it a controller, invent a control language, or just use Python and tool calls, and attach each tool to its own harness. Talk, Implication 3 Talk

Three claims sit inside this. In-context learning comes free of robot data. Reflexes can be trained into the language model itself instead of living in a hand-built controller. And the interface between the model and the robot is a design choice that can be written down. Talk The first is now measured, with a catch: free of robot data is not free of design. The same model got worse with naive demonstrations (Part 03) and much better with a structured context.

RoboICL's grammar

RoboICL (Liu et al., arXiv 2609.34261) gives the model one action tool. The model emits a continuous action chunk and a short note; deterministic code validates it, solves inverse kinematics and executes the admissible prefix. Write-up Everything in context, whether a recorded demonstration or the model's own last attempt, is written in one grammar: observation, action chunk, execution receipt, result observation. Sampled demonstration blocks sit beside a bounded memory with fixed anchors, so early experience survives later stages, and the latest interaction is always present for immediate correction. When steps are discarded, an explicit <TRAJECTORY_GAP> record stops the model reading two blocks as consecutive. Measured

On two development tasks, an optional Jev-gated action reuse cut Astra calls by 33 to 48%, which matters for Part 09's speed budget. On a ten-task subset RoboICL with Astra alone scores 60.60 against 62.60 reported for a π0.5 plus Astra ensemble. Measured

Tools as backends: AgentSTAR

AgentSTAR (Mazur, Karaev, Chang, Malik and Shafiullah, arXiv 2609.24487) reconstructs and tracks articulated objects from one monocular video. A vision-language agent refines shape and generalized pose in a render-and-compare loop, mixing coarse visual reasoning with numerical pose optimisation, and the output is a structured 3D object model with parts, joint axes and pivots. It beats 3D point-tracking baselines for articulated objects on ARCTIC and every evaluated rigid-object tracker on HOT3D. Measured The discussion's point is the role it could play: a kinematic-replay backend that an agent calls when it needs an object's geometry, the way a coding agent calls a compiler. Talk

Low-level control without chain-of-thought

Low-level control does not need chain-of-thought, and nothing stops a language model from skipping it. Give the output format room to reason when needed and to act directly otherwise. Discussion Talk

Rotations, simple contacts and kinematic primitives are intuitive: they need neither reasoning nor tokens spent on it. The continuous dynamics underneath may need a JEPA-style model, which the discussion did not see as a contradiction of the thesis. There is no need to inject these skills into the weights; the output format can force the choice. Talk

Shipped systems already make this choice. π0.5 decodes discrete tokens for high-level steps and continuous flow-matching outputs for motor commands, in one model. Measured In the Galbot hybrid, Astra passed π0.5's actions through and rewrote only 14.4% of control steps. On RoboCurve's bowl task Astra spent 2.1k output tokens per run against 12.9k for Claude Fable 5.1, and succeeded 19 times to 8. That is a cross-model comparison, so it does not isolate thinking, but heavy token use was clearly not required. Demo

Two architecture pointers

RoboJEPA as a residual. The discussion proposed placing a RoboJEPA-style world model beside the language model to absorb the low-level dynamics the language model should not reason about. The RoboJEPA paper itself has no such module; "residual" is the discussion's proposal. Talk

Attention Residuals. Kimi's AttnRes (arXiv 2603.15031) replaces the fixed, unit-weight residual sum of PreNorm transformers with softmax attention over all earlier layer outputs, so each layer chooses which depths to read with learned, input-dependent weights. Block AttnRes attends over block summaries to save memory. In Kimi Linear (48B total, 3B active) pre-trained on 1.4T tokens it improved every evaluated downstream task. Measured The write-up's reading of why it came up: if each layer can choose its depth, shallow intuitive features and deep reasoning features can live in one network. Write-up

One system or two

A System 1 and System 2 split makes sense as an intermediate step. If end-to-end is possible, you do end-to-end. Talk Talk

The boundary is blurrier than the labels. Figure's Helix runs a 7 to 9 Hz vision-language model as System 2 and a 200 Hz visuomotor policy as System 1, and trains the pair end-to-end. Measured So the useful question is not one system or two but whether the interface between them is learned or hand-designed. Click a system to place it on that axis.

Current data cut both ways. On bimanual RoboDojo tasks the π0.5 plus Astra hybrid beat Astra alone, 48% to 26%. On simpler single-arm RoboLab tasks Astra alone won, 98% to 92%. Write-up

Solve it big, then distill

Solve general intelligence with a large model, then distill it. Distilling a capable large model beats fine-tuning a small one directly. Talk Talk

The best public evidence comes from language, not robots. DeepSeek-R1 (arXiv 2501.12948) trained the same Qwen-32B base two ways: large-scale RL directly, and distillation from the full R1 model. Distillation won on every benchmark in Table 6. Measured

The paper's own caveat matches the talk's order of operations: going beyond this may still need stronger base models and larger-scale RL, so you solve it big first and distill second. Robotics has no controlled equivalent of Table 6. Reviewer

The next concrete step the talk named is a GPT-7-nano class model on better chips, and the reason is speed (Part 09). That raises the real question: which abilities go to the edge? Talk The write-up proposes a split by control rate. Move the abilities yourself and see which placements break the loop.

More structure, less scalability

The more structure a method has, the less it scales. Structure added for today's compute becomes tomorrow's ceiling. Talk, closing slide Talk

The slide's plot carries no data. It restates two older positions: Hyung Won Chung's argument (Stanford CS25, 2024) that the right amount of inductive bias depends on the compute available, and Rich Sutton's bitter lesson.

The talk's synthesis: brute-force token projection in the style of ViT, plus agentic loops, plus distillation, outperform heavily engineered inductive biases once compute and verification exist. On this reading language does not replace physics or hardware. It eats the engineering layers that stood between language-scale intelligence and physical competence, slowly at first and then suddenly. Talk

Where the thesis is weakest

Every hot take has a measurable version, and none is settled. The table pairs each with the strongest counter-evidence available today and the experiment that would settle it. The first two columns follow the write-up's own audit; the reviewer notes below them are mine.

What I would add as a reviewer

1. One model carries the case. Nearly every robot result is GPT-6 Astra within five weeks of release, as preprints, leaderboard entries and demos. GPT-5.5 at 0.88% on RoboDojo shows that "language models" is the wrong unit of analysis. Reviewer
2. The metrics do not line up. RoboICL reports progress scores, RoboDojo success rates and Scores, ENPIRE pass@8, the Galbot report reweighted baselines on tasks chosen for low baseline success, and DeepSeek-Flash ran 10 episodes per task against 50 for the others. A sentence that compares two of these numbers needs a protocol flag. Reviewer
3. Safety is not a column. The only real-robot benchmark attempt with Astra was halted after unsafe actions damaged hardware. None of the twelve takes is stated with a safety metric, yet interventions per hour is the number a deployment needs first. Reviewer
4. Demo lists are survivor lists. Collections of community demos record what worked. Without the attempts that failed, a demo shows possibility, not rate. Reviewer

Experiments, ranked by what they buy per unit of effort

What this changes for my research

My work sits on three of the talk's fault lines: world models, agentic policies and the representation a manipulation policy needs. Here is where the argument moves my plans and where I think it is wrong.

World models: a fitted simulator is one

The talk's real2sim objective is the training loss of an action-conditioned world model, with a physics engine as the function class. For the latent world models I build for mobile robots and for latent lookahead in manipulation, the useful import is the evaluation: score rollouts over a horizon and score contact events separately, and keep dynamics fidelity, controllability and downstream planning utility as three axes. RoboJEPA's claim that imagination error predicts planning success is a hypothesis I can test on my own models before trusting it. Reviewer

Agentic policies: keep a grounded low level, and report recovery honestly

The hybrid results (48% against 26% on bimanual tasks) support what AFFORD-X and the AGI latent-lookahead work already assume: a reasoning model selects and verifies, and a grounded layer handles metric precision. RoboICL's grammar of observation, action, receipt and observation is a concrete template for an agent's memory of its own attempts. And Lab A is a warning for my own loop numbers: a loop that retries must be reported at pass@1 as well. Reviewer

Representation: the outcome decides what is noise

"Exact kinematics are redundant" is the bet behind learning manipulation-sufficient representations through outcome bottlenecks, put in other words: keep what changes the outcome and drop the rest. The insertion counter-case in Part 07 is the test such a representation has to pass. Reviewer

Your verdict

Your credences from Part 02, your answers to the talk's three open questions, and eight questions to check the lecture landed. Everything stays in this browser.

Your credences

The three questions the talk left open

Check yourself

0 of 8 answered

Sources and glossary

Glossary

Click a term for the example from this lecture.

Sources

Built by Md Selim Sarowar on 12 Oct 2026 from Junfan Zhu's write-up of Tonghe Zhang's talk at Saturday Robotics #33. The prose, figures and labs are my own; the talk's claims are paraphrased and attributed. The numbers graded Measured were checked against the cited arXiv papers that day (RoboDojo Table 5 and the 340-pair ICL table, RoboICL, VIRAL, ENPIRE, RoboJEPA, AgentSTAR, Attention Residuals, the VLN-CE report, Onshape's limits page). Numbers graded Write-up come from the article and were not re-checked. Labs are illustrations with stated assumptions, not reproductions.