An AI that can produce photorealistic videos of sign language interpreters from speech could improve accessibility by removing the need for humans.
Ben Saunders at the University of Surrey, UK, and his colleagues used a neural network that converts spoken language into sign language. The system, called SignGAN, then maps these signs on to a 3D model of the human skeleton.
The team also trained the AI on videos of real sign language interpreters, teaching it how to create a photorealistic video of anyone signing based off an image of them. By combining this video with the 3D skeleton, the AI is able to covert spoken words into sign language.
Advertisement
One of the key challenges was accurately generating realistic hand images, given sign languages rely on the positioning of fingers to differentiate words. Since the signers’ hands in the training videos could sometimes be blurred, the researchers used an existing AI capable of estimating hand poses from a small patch around the middle knuckle to avoid the system learning to generate blurred hands.
They tested the end result by showing 10-second generated clips to 46 people, around a quarter of whom were signers. All of them said they preferred SignGAN to other AI models. The researchers declined to comment on the work.
“This technology is about making realistic images of signers generated from input, which is welcome,” says Adam Schembri at the University of Birmingham, UK. He also praised the model for making the signing avatars look more natural and less robotic.
Liam O’Dell, a Deaf journalist and campaigner in the UK, also welcomed the work. “Having seen many gloves which have tried to convert sign language into spoken language – often with low levels of success – seeing a proposal for technology which works the other way around is rather refreshing,” he says.
But sign language interpretation is a skilled job that may not be ready to be replaced by AI. “It’s important that any future development of this work appreciates the many complex elements which make up sign language – such as facial expressions, mouth movements and context – and is co-produced with Deaf people,” says O’Dell.
Reference:
