Robot training data: why there is no internet of actions

Ask a large language model how to load a dishwasher and it will give a competent answer: plates facing the centre, glasses on the top rack, nothing blocking the spray arm. Ask a robot arm to do it and you discover that the model’s answer is the easy part. What the arm needs is not the plan but the motion: where to put two fingers on a wet plate, how hard to squeeze, what to do when the plate slides a centimetre during the lift. That knowledge exists, in billions of human hands. It has simply never been written down.

This is the quiet asymmetry behind the current wave of robot learning. Language models were built on a resource that already existed. Decades of people writing for each other produced a corpus of text that only needed to be collected, filtered and fed in. There is no equivalent pile for physical action. No archive records, joint by joint and newton by newton, how anyone has ever picked up a cup. Every example a robot learns from has to be produced on purpose, and producing it is slow, expensive and tied to a specific machine.

What a robot actually learns from

The dominant recipe in 2026 is a family of models called vision-language-action models: a network that takes a camera image and a written instruction and outputs motor commands. Their architecture borrows heavily from language models, and so does their appetite. They improve with data, and the relevant data is not images or sentences. It is the pairing of what the robot saw with what its joints did next.

The standard way to produce that pairing is teleoperation, meaning a person drives the robot remotely, often through a second arm or a handheld controller that the robot mirrors, while every camera frame and joint angle is logged. The result is a demonstration: a recording of a task done correctly, from the robot’s own point of view, in the robot’s own units. Imitation learning, the technique of training a policy to reproduce those recordings, then turns a pile of demonstrations into behaviour.

The trouble is the arithmetic of a human operator. One person, one robot, one task at a time, in real time. Teleoperation scales linearly with paid hours, and nothing in the process gets cheaper as you do more of it.

How big the biggest piles are

The public reference points show the scale of the problem. DROID, released in 2024 by a consortium of labs, is one of the most diverse real-world manipulation datasets ever gathered. Its authors report 76,000 demonstration trajectories, or about 350 hours of interaction, collected by 50 operators across 564 scenes on three continents over twelve months. A year of coordinated work by dozens of people, and the result would fit in two weeks of continuous video.

The other landmark, Open X-Embodiment, took the opposite route: rather than collecting, it pooled. The collaboration gathered more than a million trajectories from 22 different robot types across 21 institutions. That sounds large until you look at what the trajectories are. Many are short, many come from simple grippers on tabletops, and they were recorded on machines with different arms, different cameras and different joint layouts. A trajectory from one robot is not directly an instruction for another. Pooling helps, and the paper showed that models trained on the mixture beat models trained on each lab’s data alone, but it does not erase the boundaries between bodies.

That second point is the one that makes robot data structurally harder than text. A sentence written in 1950 is still a valid sentence. A demonstration recorded on a robot that has since been redesigned is a demonstration for a body that no longer exists. Every new hand, every new arm length, every new camera position devalues part of the archive.

The detour through human hands

If the bottleneck is that robot hours are expensive, the obvious move is to stop paying for robot hours. Humans already perform every manipulation task worth automating, all day, and cameras worn on the head can record it from roughly the angle a robot would see. The question is whether video of a person doing something can teach a machine to do it.

The difficulty is that video records outcomes, not commands. A clip of a hand closing on a mug shows where the fingers went. It does not show the motor torques, and it shows them for a hand with five fingers and twenty-odd degrees of freedom, a count of the independent ways the joints can move. Turning that into actions for a robot hand requires two steps: estimating the human hand’s pose in three dimensions from the image, then retargeting it, the term for mapping one body’s motion onto another’s joints. Both steps introduce error.

The bet of 2026 is that volume beats that error.

Where this stands in 2026

Three recent preprints, none of them yet peer reviewed, sketch how far that bet has got.

The largest is EgoScale, posted in February by a group spanning industry and academia. Its authors trained a vision-language-action model on more than 20,854 hours of egocentric human video with action labels, which they describe as over twenty times larger than prior efforts, then fine-tuned it on a robot with a 22-joint dexterous hand. They report a 54 percent improvement in average success rate over the same policy trained without the human data. More interesting than the headline is the shape of the curve: the authors report a log-linear relationship, meaning that each tenfold increase in human video lowered the model’s error by a roughly constant step, and that this error tracked real-robot performance. That is the signature that made language models predictable to scale. Seeing it in robot learning, if it holds, would let a lab estimate in advance what a given investment in video will buy.

The second, Zeva-Ego, posted in September by a group from Tsinghua University and an associated institute, tries to put a price on the substitution. Scaling egocentric data to 10,000 hours raised success on RoboTwin, a simulated dual-arm benchmark, from 63.8 to 75.3 percent, which matches the 74.7 percent obtained from 2,000 hours of robot demonstrations. The authors summarise this as an exchange rate of roughly four to five hours of human video per hour of robot data.

The third, HumanEgo, goes the other way and asks how little human video is enough. Its authors report 92.5 percent average success across four real-world tasks from thirty minutes of human video per task, and 75 percent from fifteen minutes, beating an equal amount of robot teleoperation time by 41 percent. Their trick is to throw away most of what the video contains and keep a compact description of how the hand and the object move relative to each other, which travels between bodies better than raw joint angles.

Put side by side, these results describe a field converging on an inversion. For most of the last decade, the robot’s own experience was treated as the primary data and human video as a curiosity. The 2026 recipe treats human video as pre-training, the broad and cheap foundation, and robot demonstrations as the expensive final layer that teaches a particular body to use what it has learned. Industry has followed: teleoperation rigs and data collection pipelines are now sold as products, and browser-based schemes that let remote volunteers drive robots are being tested as a way to crowdsource the last layer.

What the video cannot see

There are good reasons not to treat the exchange rate as settled.

The first is that video is blind to force. A camera sees fingers close on a mug. It does not see that the grip was light enough not to crush a paper cup and firm enough not to drop a full one. For most pick-and-place work, position is enough. For the tasks that motivated humanoid robots in the first place, threading a cable, peeling a sticker, turning a stiff valve, the information that matters is in contact forces, which a head-worn camera does not record. Gloves with pressure sensors exist, but strapping them onto thousands of people undoes much of the cheapness that made video attractive.

The second is that demonstrations are curated. People recording themselves tend to do the task well, so the archive is full of success and nearly empty of recovery: the plate that slipped, the drawer that stuck, the second attempt. A robot trained mostly on clean executions has seen very few of the situations where it most needs guidance. Teleoperation has the same bias, but a robot logging its own failed attempts can at least fill the gap. A video archive of humans cannot.

The third is measurement. The Zeva-Ego exchange rate comes from a simulated benchmark, and a ratio measured in simulation says little about how the same trade plays out on a real arm in a cluttered kitchen. HumanEgo’s figures come from four tasks. EgoScale’s scaling curve correlates a training loss with robot success, and the strength of that link is the authors’ own assessment. All three are preprints. None of this makes the results wrong, but none of it is yet the kind of evidence that let language model scaling laws become an industrial planning tool.

The last is the boundary between bodies, which human video narrows but does not remove. Retargeting a human hand onto a three-finger gripper discards most of what the hand did. The richer the robot hand, the more of the human signal survives, which quietly favours humanoid designs: not because a human shape is mechanically ideal, but because it is the shape the cheapest training data was recorded on.

That last point may be the most consequential. If the dominant data source for robot learning is footage of people, then the robots that learn fastest will be the ones that most resemble the people in the footage. The frontier between human and machine is being drawn, in part, by what was easiest to film.

Further reading

• Open X-Embodiment: Robotic Learning Datasets and RT-X Models, the pooled dataset and the first evidence that mixing robot bodies helps.
• DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset, a detailed account of what a year of real-world collection produces.
• EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data, the largest human-video pre-training run to date and its scaling curve.
• Zeva-Ego: Egocentric Mid-Training with In-Context Causal Learning for Robot Manipulation, where the human-to-robot exchange rate is measured.