Robot training

Teaching a machine to use its hands

The short answer

Robot training teaches a machine how to move its arms and hands to get a job done. For years that meant a person steering the robot by remote, one task at a time. Now teams also learn from video of people doing the real work.

First-person view of a cleaner wringing a string mop in a yellow bucket, both hands in view.

1You recordA cleaner wears the phone and does the normal job.
2You uploadThe video goes up through the recording app.
3We check itOur team reviews quality within 24 to 48 hours.
4It gets labeledTasks, hands and tools are marked in the video.
5AI and robots learnTraining teams use many hours of it to teach their systems.

The slow way

The older method is to have a person steer the robot through a task by remote, over and over. Wipe the counter. Again. Again. Each run is saved and used to teach it.

That is slow and costly. One robot in one lab only gets through so many counters in a day, and every lab looks about the same.

Learning from people instead

People already clean millions of rooms each night. If a robot can learn from video of that work, the pile of examples gets much bigger, fast.

Researchers at Apple built a set called EgoDex with 829 hours of first-person video across 194 tabletop tasks. They wrote that robot hand skills have a data problem: there is no internet-sized pile of this kind of video yet.

Figure has shown hand work as well. In early 2026 it showed its Helix 02 robot unloading a dishwasher across a full kitchen, a four-minute job with no help. Figure said part of that system was trained on over 1,000 hours of human motion data.

From your video to a robot

A cleaner records a real job, like mopping a hallway, with the phone on their forehead.
The video is checked, then marked up: where the hands are, what the tools are, what step is happening.
A robot team trains its model on many hours like it from many buildings.
The robot practices, often in a computer copy of a room first, then in a real one.

Why this matters to your crew

A robot has hands too, but it has never held a mop. It learns hand skills from seeing hands. That is why your crew's full palm and all fingers in the frame make an hour worth more to these teams, and why a mounted phone that slips down to the floor has to be fixed before recording goes on.

What other owners ask

Can a robot learn only from video of people?

For some skills, teams have reported yes. Figure said its robot learned to find its way around homes from first-person human video alone. Hand skills like wiping and wringing are harder, and most teams mix people's video with robot practice.

Does Colby build robots?

No. We pay cleaning companies to record real work. The video helps AI and robotics teams learn how cleaning is done.

Where to read more

These are public examples from other companies. They are not Colby partners or customers.