Last updated 26 September 2026
1. Introduction
In this article, we take a look at how humanoid robots learn through videos that show humans in action. A growing number of companies now record everyday human tasks and use that footage as training data, to supplement or reduce reliance on the generally slower or more expensive alternatives: teleoperation (remote control), simulation, or letting the robot learn through its own trial and error. Although video alone isn’t the whole story, and real experience still matters, videos can teach a robot a lot.
2. The training roadmap
The process is roughly: record video of people doing things, then remove repeated or unusable footage, leaving a cleaned-up set of video (this is the dataset). This dataset is used to train the model, the AI system that learns from data and will eventually control the robot. This first round of training, done on the dataset before the model ever touches a real robot, is called pretraining. Pretraining is how the model identifies useful patterns in the video: how objects move, how hands grip things, none of it specific to any one robot yet. After that comes adaptation: the model gets trained further on one particular robot, so it learns that robot’s actual body and sensors. Once adapted, the robot is deployed, and it starts generating its own experience too, which can be used for further training. This is separate from the control system that runs while the robot is actually moving (Section 8).
3. What video can teach
A video teaches a robot a lot about how humans move: which direction a hand travels, roughly how fast, what order the steps happen in, and how the body positions itself. What it can’t do is copy a human’s movement exactly, angle for angle, because a robot’s arms and joints usually aren’t built the same way a person’s are. A video also shows the bigger picture of the task itself: which objects the person used, where they made contact, and what the finished result looked like, even though a robot never gets the human’s actual experience, just a video of it. One recent experiment shows how far that video alone can go.
Example: Hand–object geometry (HumanEgo, University of Maryland, June 2026).
A team at the University of Maryland wanted to see how little human video it would take to teach a robot a task. A person wore smart glasses and recorded themselves doing a task with their hands, no robot involved at all. Their system then digitally erased the human arm from the video and represented the hands and objects through their relative positions and orientations. They fed that into a robot, with no robot-specific task training. With just 30 minutes of this video per task, the robot succeeded 92.5% of the time on four real-world tasks.1,2
4. Working Out a Path from Video
A video shows what a person did, but a robot still needs something more specific: a path, a plan for how to actually move. Some methods extract the path directly from the video. Others use the video only for a hint or a goal, then let the robot work out its own path through trial and error. The two examples below show each approach.
Example: Hand trajectories from ordinary video (VidBot, 2025). Researchers at the Technical University of Munich and ETH Zürich built VidBot, a system that watches ordinary video and works out the path a person’s hand took, then turns it into a movement a robot can follow. In simulation, it scored well ahead of the next-best method across 13 tasks. It was also tested on a real robot, in a separate evaluation.3
Example: A video that supplies a reward (Human2Sim2Robot, 2025). Researchers at Stanford University built Human2Sim2Robot, which pulls two things from a single video: the path the object followed, and the hand’s pose just before manipulation began. The robot uses those two clues as a guide and works out its own path through trial and error in a computer simulation. From just one video, it beat one baseline that simply replayed the recorded motion by more than 55%, and a standard imitation-learning baseline by more than 68%.4
5. From Path to Motor Command
Even once we know the path a hand or arm should follow, a robot still needs to convert that into motor commands for every joint involved: how far to turn and how much force to apply.
One common technique for the “how far to turn” part is called inverse kinematics, though not every system solves it this way: working backward from where you want a hand, foot, or any other part to end up, to figure out the angle every joint along the way needs to bend, shoulders, elbows, knees, and so on. This can run continuously, recalculating as the target moves.
Angles alone aren’t enough, though. A motor still needs to know how hard to push, which depends on forces: the weight of the robot’s own arm, friction in its joints, and whatever resistance it meets at the object, how much grip it takes to hold something, or how hard a surface pushes back. This is also where video reaches its limits. It can show someone gripping a cup, but not how hard they gripped it or whether it slipped. A robot has to sense that itself.
Cameras struggle here too, in head-mounted video the hand often blocks the camera’s own view of the contact point. And a robot’s physical limits show up again: a path that works for a human body might ask a joint to bend further, or move faster, than it’s built for.
Example: Learning contact through simulation. In 2025, researchers at Carnegie Mellon University built a system called HDMI, short for “HumanoiD iMitation for Interaction.” It takes human and object movement from video, then has a robot practice separately in a physics simulation, learning contact and balance through trial and error, before transferring straight to a real robot with no extra training in between. Tested on an actual Unitree G1 humanoid, it managed 67 consecutive door traversals and six tasks combining walking with handling objects. The video showed what should happen; the contact and balance behavior was learned through reinforcement learning in simulation.5
6. Robot-Side Experience Is Still Mostly Needed
Even after everything video can provide, robots still mostly need experience of their own. Human video can reduce how much robot-specific data is needed. HumanEgo above shows it can sometimes remove the need for robot-specific task training entirely, for a narrow set of tasks. But usually, some robot-specific data is still needed to teach things video can’t: how its commands translate into actual motion, timing, balance, and recovering from mistakes.
Example: Mixing in human video (Physical Intelligence, December 2025). Researchers at Physical Intelligence added first-person human video into the extra training (“fine-tuning”) of π0.5, their general-purpose robot AI model. This roughly doubled performance on four new situations that only appeared in the human video, without needing any explicit human-robot alignment. But this only worked after the model had already been trained on a wide variety of robot scenes and tasks first. Human video was added to that foundation; it didn’t replace it.6
Example: Human motion for control, teleoperation for skills (HumanPlus, CoRL 2024). Researchers at Stanford University first trained a system in simulation on 40 hours of existing human motion data. That let a humanoid copy a person’s movements in real time using just one camera (“shadowing”). Operators then used this shadowing ability to control the robot remotely (teleoperation) and record real demonstrations. Copying those demonstrations (“behavior cloning”) taught the robot skills like folding a sweatshirt, succeeding 60-100% of the time from up to 40 demonstrations.7
Across such systems, the amount of human video used ranges from 30 minutes to over a million hours, and how much robot-specific data was still needed varies just as widely.
7. Does More Human Video Actually Help?
Two examples give a rough answer: yes, but with real limits worth knowing about.
Example: A scaling ladder (Dyna-2, August 2026, company-reported). Researchers at Dyna Robotics trained models on growing amounts of human video, from 1,000 hours up to over a million, then gave each version the same small amount of robot practice (at most 10 hours per task). Averaged across 14 tasks, performance rose steadily with more video: from 20%, to 28%, to 45%, to 53%. But individual tasks told a messier story. One task (turning a key in a lockbox) stayed stuck at 0% until suddenly jumping to 90% at the largest scale. Another (tying a rope) actually got worse, dropping from 90% to 40%. With only 10 trials per task, the reason for that reversal is unclear. And a task needing just 10 minutes of robot practice still climbed from 10% to 50% as the video grew. Every task still used some robot practice, so this shows video making a robot better at tasks it’s already learning, not a robot picking up entirely new chores from video alone.8
Example: Starting from scratch vs. starting from video (Figure Helix 2.5, 17 September 2026, company-reported). Figure trained its Helix 2.5 model on its own video dataset, then taught it three household tasks: tidying, folding towels, and making a bed. To see if the video training actually helped, they compared two versions of the robot, both given the same hands-on practice at these tasks, in 30 homes neither had seen before. The only difference was that one version started from scratch, with no video training, and the other started from the video-trained model. The from-scratch version succeeded 9% of the time; the video-trained version succeeded 56%.⁹
8. Learning from Video vs. Robot Control Systems
Everything so far, the path a robot works out from a video (Section 4), the motor commands and force sensing needed to follow it (Section 5), still has to actually run, live, while the robot moves. That’s the job of a robot’s control system, and it’s a genuinely different kind of thing from what was learned beforehand: learning from video happens during training, sometimes across several stages, but control has to happen continuously, in real time, for as long as the robot is moving.
A useful way to think about a robot’s control system is as three layers, running at different speeds, because different parts of the job happen at wildly different speeds. Deciding “pick up the cup, following this path” is a slow, thoughtful kind of decision, maybe just a few times a second (measured in Hz, or updates per second), this is roughly where a plan gets carried out. Keeping a two-legged robot balanced needs a faster layer, tens to hundreds of times a second. And reacting to force, touch, and body position needs the fastest layer of all, potentially close to 1,000 times a second, the kind of split-second correction a body makes without “thinking” about it.
9. How to Read a Reported Result
We talked about success rates above. Two things are worth knowing:
Doing well at each step doesn’t guarantee finishing the whole job. For example, assuming each step succeeds independently, if a robot succeeds at each step of a 14-step task 90% of the time, the odds of finishing all 14 steps in a row are only about 23%. Even at 99% per step, that’s still only about 87% overall. To reliably finish the whole task 90% of the time, each step needs to succeed over 99%.
Few tests means an unreliable number. For example, a reported “80% success” tested only 50 times could really be anywhere from about 69% to 91%. Tested 400 times, it’s more solid, likely 76% to 84%. If the tests are highly similar or not independent, the real uncertainty can be larger than this simple calculation suggests.
10. Where Does All This Video Actually Come From?
Not all human video is equally useful, raw hours of footage aren’t a good measure of value, what matters is how much gets properly filtered down to relevant, usable material. Thousands of hours from a thousand different rooms doesn’t automatically teach a robot a thousand different skills. At its August 2026 launch, Figure said its Index app was receiving 30 minutes of video every second (about 43,200 hours a day, by its own reporting), with over 16 million videos uploaded and $15 million paid to contributors.10
Getting this data is also expensive and labor-intensive, real people are the ones actually filming everyday tasks, often in dozens of countries. In one case reported by MIT Technology Review, a worker was paid $15 an hour.11 And interestingly, Spirit AI told Reuters that using varied, “dirty” data with a wider range of motions helped its models improve faster.12
There’s a simple way to think about the cost of all this: cost per useful hour equals the price per recorded hour, divided by how much of that footage actually gets kept after filtering. For example, $10 per recorded hour with 40% of it usable works out to $25 per useful hour.
Human video is one option among several for gathering training data, alongside having a robot practice in simulation, having a person directly control it remotely, or having it learn through its own real-world trial and error. Each comes with its own costs and tradeoffs, and none of the sources here directly compare them head to head.
11. Safety, Privacy, and Standards
Robot safety features, like limits on how hard a robot can push against something, or systems that stop it if it starts to fall, are built directly into its hardware and control software. They aren’t something a robot picks up from watching videos. People don’t always do things the safest way in everyday life, so a robot copying ordinary human habits could end up copying bad habits, not just good ones. Official safety standards for human-like robots don’t fully exist yet, the relevant international rules are still being written.
Home video also raises privacy questions, it often captures other people in the background, children, personal documents… not just the person doing the task. It’s not always clear how that footage ends up being used, shared, or stored, or whether people filming it fully understand where it goes.
Conclusion
The goal is not perfect imitation. It’s giving a robot a reasonable starting point so it isn’t rediscovering everything about the physical world through costly trial and error alone. Human video is one way to supply that starting point, a broad prior about tasks, objects, and motion. But video only captures part of how humans actually learn physical skills, we also learn a great deal through direct trial and error, feeling the 3D world with our own senses, not just by watching. Either way, a robot still needs the force, contact, balance, and control that video alone can’t teach, learned through its own practice, whether that’s in the real world or in simulation.
Sources
- HumanEgo (arXiv 2605.24934)
- University of Maryland announcement of HumanEgo (3 June 2026)
- VidBot (CVPR 2025): arXiv abstract · full text (PDF) · CVPR page
- Human2Sim2Robot (CoRL 2025)
- HDMI (arXiv 2509.16757)
- Physical Intelligence: human-to-robot transfer (Dec 2025)
- HumanPlus (CoRL 2024)
- Dyna Robotics: Dyna-2 (Aug 2026)
- Figure: Helix 2.5 (17 Sept 2026)
- Figure: Introducing Index (25 Aug 2026)
- MIT Technology Review: gig workers training humanoids (1 Apr 2026)
- Reuters: Spirit AI interview (18 Sept 2026)
Post By: A. Tuter
Terms of Use: Unauthorized copying is prohibited; we maintain dated records to document original publication. Content may contain inaccuracies. All trademarks mentioned belong to their respective owners; we are not affiliated with them or any other entities mentioned, and no endorsement is implied. See our Terms Page.