Google Announces Sign Language To Text Model for Pixel 11

I’m late to this, but last month Google’s DeepMind Sign Language Team announced Sign Language To Text (SL2T), which the company described as "a massively multilingual sign-language-to-text translation model” that “marks a breakthrough in quality and generality.” The feature is available on Google’s Pixel 11 phones, in Gboard and Live Transcribe, in American Sign Language at launch, with “more devices and additional languages” to come.

"Similarly to how hearing users can use dictation to speak instead of typing, [Sign Language To Text] enables Deaf users to sign to their phone anywhere they’d normally type,” Google wrote in its blog post published on August 12. “You can sign to search the web, draft messages or documents, and ask Gemini to solve queries or execute tasks. In Live Transcribe, you can sign responses in conversations instead of having to type back and forth. According to our testers, signing in ASL is faster, more natural, and more delightful than typing in English.”

The project, with more than 100,000 hours’ worth of data spanning over 50 signed languages, was conceptualized by Sam Sepah, a Deaf employee at Google. I interviewed him in May 2023 about a similar initiative involving AI and ASL.

As Google explained, sign languages matter not merely linguistically—they matter culturally too.

“Sign languages are the primary languages of Deaf communities around the world and the cornerstone of Deaf cultural identity. There is great diversity among deaf people in terms of their level of proficiency in signing, speaking, reading, and writing, so it is important to support access in all modalities,” the company said. “Deaf people can benefit from sign language processing in the same way that hearing people benefit from spoken language processing, plus the technology opens new possibilities for bridging the communication gap between Deaf and hearing communities. Despite this opportunity for positive social impact, progress in sign language AI has been slow—both because building AI for sign languages presents complex challenges and because widespread misconceptions exist about how the languages themselves work.”

The Pixel 11 line was announced last month with an event in New York City.