New AI-based game teaches families the basics of American Sign Language

Singing along to the ABC’s is one of the first lessons we get as kids (whether or not you stick to the original or new-fangled version). 

For families with deaf and hard of hearing children, that lesson isn’t any less important, with visual languages replacing auditory stimulus as a crucial part of early development. 

Digital creative studio Hello Monday, in collaboration with the American Society for Deaf Children, wants to address this through its new online game, Fingerspelling.xyz. Using machine learning, the game hopes to provide families with the building blocks of a signed language. 

“We created this fingerspelling tool with Hello Monday to help parents support their child’s mastery of sign language, and so parents can share the joy of communicating and connecting with their deaf child,” said Cheri Dowling, director of outreach and programs for the American Society for Deaf Children.  

The game is a free, browser-based app accessible to anyone with a computer and webcam. The technology tracks the users hand movements to teach them the ABCs (known as fingerspelling) of American Sign Language (ASL) in a competitive game format: Over three levels, players are challenged to make correct hand motions as quickly as possible, racking up points for accuracy. 

Anders Jessen, founding partner of Hello Monday, says the game is a fun and playful take on traditional learning. “The game leverages advanced hand recognition technology, matched with machine learning, to give you real time feedback via the webcam for each sign and word you spell correctly,” he explained. The AI creates an in-game 3-D model of the player’s hand. Based on points along the hand and finger joints, the tech recognizes each attempt at a letter and scores you for accuracy. While it was designed with parents in mind, as a resource to teach their own children ASL, fingerspelling.xyz is simple enough for children to practice fingerspelling as well. 

Let the AI know if you’re left or right handed and choose between four different levels to start practicing your hand placements. 

The game will give you ten words to master. You’re timed and get points for accuracy, so try to line up your hand as closely as possible with the example on the screen (this might take some finessing to make sure hand is fully visible). As you successfully spell each letter, the game shows your accuracy score from zero to 100 percent. After you spell all ten words, move to the next level or try to beat your score. 

The organizations hope the new tool will incentivize hearing parents and families to begin teaching themselves and their children the fundamentals of language, empowering deaf children to communicate before they have access to ASL courses in school settings. It provides families with the opportunity to learn an “essential part” of ASL as soon as possible, the game’s creators explain.

The American Society of Deaf Children explains that 90 percent of deaf or hard of hearing children are born to hearing parents. Many of these families have never communicated or even been exposed to American Sign Language, a barrier than can frequently leave these children behind in language and other cognitive skills. 

“Without being introduced to sign language at an early stage, a deaf child may miss out on learning language. This can lead to language delay or deprivation, which has long-term negative impacts on a child’s life,” a representative of Fingerspelling.xyz explained. The National Association for the Deaf says that early access to a visual language like ASL helps children’s attention, vocabulary, and more. 

Free, online tools like Fingerspelling.xyz encourages this exposure — hopefully empowering parents to build multi-lingual households and giving children visual skills to communicate with their surroundings. In line with this, Jessen says that after mastering fingerspelling.xyz, the next step for parents of deaf children is to sign up for in-person or online ASL courses. 

For more digital ASL resources, the American Society of Deaf Children also offers paid online courses, like ASL book readings, beginner, intermediate, and advanced ASL courses for parents and children, and other educational talks. The organization also has a directory of children’s stories translated into ASL, free to download, as well as an interactive map of state-specific ASL resources, such as early intervention and youth programs.