A soft faced humanoid robot has learned to move its lips in sync with speech and song simply by watching people talk on video and studying its own reflection in a mirror.
The machine, called EMO, was built at Columbia Engineering and is at the center of a new study showing thatrobots can pick up complex speech related gestures through observation instead of hand written code.
The work, which appears in the journal Science Robotics, points to a future where robot conversations feel far less stiff and cartoonish than they do today.
The work, which appears in the journal Science Robotics, points to a future where robot conversations feel far less stiff and cartoonish than they do today. The work, which appears in the journal Science Robotics, points to a future where robot conversations feel far less stiff and cartoonish than they do today.
The work, which appears in the journal Science Robotics, points to a future where robot conversations feel far less stiff and cartoonish than they do today.
Why faces and lips matter in conversation
If you have ever found yourself staring at someone’s mouth while they speak, you are not alone. Eye tracking studies suggest humans devote a notable share of their attention to lips and lower face during conversation, which is one reason clumsy mouth motion makes many robots feel unsettling. EMO tries to solve that.
Its silicone face is driven by 26 tiny motors that can pull and push the lips with fine control, more like human muscle than the rigid jaws seen on many social robots.
How EMO trained itself using a mirror and YouTube
Training started with a kind of robotic mirror play. Engineers sat EMO in front of a reflective surface and let it fire off thousands of random expressions while a vision to action model learned how different motor patterns produced different mouth shapes.
Once the system understood its own face, the team fed it hours of talking and singing clips on YouTube. By matching the sounds it heard with the lip positions it saw, the robot gradually learned to turn raw audio into the right sequence of facial movements across ten different languages.
What people noticed in tests
To find out whether people actually bought the effect, the researchers showed videos of EMO speaking to more than one thousand volunteers.
Viewers compared three different control methods against a reference of ideal lip motion and chose the new vision to action approach in roughly sixty two percent of trials, far ahead of the simpler baselines that only tracked loudness or copied past examples.
Hard consonants such as B and sounds that require lip puckering still trip the system up, but the team expects performance to improve as EMO keeps “listening” and practicing.
Why this could change human robot interaction
For the most part, the bigger story is what happens when this kind of realistic face is paired with conversational artificial intelligence.
Lead author Yuhang Hu notes that combining fluent lip syncing with modern dialogue models could make exchanges with robots feel more like talking to another person than to a machine, especially in settings such as classrooms, hospitals, or elder care homes where empathy and trust matter.
That possibility cuts both ways. Study supervisor Hod Lipson has warned that robots which smile and speak convincingly will be powerful tools and should be developed slowly and carefully so they help people without misleading them.
If billions of humanoid machines are coming, as some economists suggest, then teaching them to “use their face” responsibly may matter as much as teaching them to walk.
The study was published on Science Robotics.
