Enabled Play accessibility device turns faces into game controllers

0

Comment

Over the decades, input devices in the video game industry have evolved from simple joysticks to sophisticated controllers that emit haptic feedback. But with Enabled Play, a new assistive technology created by self-taught developer Alex Dunn, users are embracing another type of input: facial expressions.

While companies like Microsoft have sought to expand accessibility through adaptive controllers and accessories, Dunn’s new device takes those efforts a step further, translating users’ head movements, facial expressions, real-time speech and other non-traditional input methods in mouse clicks, keystrokes and joystick. movements. The device has users raising their eyebrows – literally.

“Enabled Play is a device that learns to work with you — not a device you have to learn to work with,” Dunn, who lives in Boston, said via Zoom.

Dunn, 26, created Enabled Play so everyone, including his disabled younger brother, could interact with technology in a natural and intuitive way. At the start of the pandemic, the only thing he and his brother from New Hampshire could do together, despite being about 70 miles apart, was gambling.

“And that’s when I started to see firsthand some of the challenges that he had and the limitations that games had for people with really any type of disability,” he said. -he adds.

This geographer helps video game developers avoid angering countries

At 17, Dunn dropped out of Worcester Polytechnic Institute to become a full-time software engineer. He started researching and developing Enabled Play two and a half years ago, which initially proved difficult because most speech recognition programs lagged in response time.

“I built prototypes with voice commands, then started talking to people who were deaf and had various disabilities, and found that voice commands weren’t enough,” Dunn said.

It was then that he began to think outside the box.

Having already built Suave Keys, a voice-activated program for gamers with disabilities, Dunn created Snap Keys – an extension that turns a user’s Snapchat lens into a controller when playing games like Call of Duty, “Fall Guys” and “Dark Souls”. In 2020, he won two awards for his work at Snap Inc.’s Snap Kit Developer Challenge, a competition between third-party app makers to innovate Snapchat’s developer toolkit.

With Enabled Play, Dunn takes accessibility to the next level. With a wider variety of inputs, users can connect the assistive device – equipped with a robust processor and 8GB of RAM – to a computer, game console or other device to play games in the way that suits them best.

Dunn also spent time making sure Enabled Play was accessible to people who are deaf, as well as people who want to use non-verbal audio input, such as “ooh” or “aah”, to perform an action. Enabled Play’s vowel detection model is based on “The Vocal Joystick”, which engineers and linguistics experts at the University of Washington developed in 2006.

“Essentially it seems to predict the word you’re going to say based on what’s in the profile, rather than trying to assume it could be any word in the dictionary,” Dunn said. . “This helps reduce machine learning bias by learning more about how the individual speaks and applying it to the desired commands.”

Dunn’s AI-enabled controller takes into account a person’s natural tendencies. If a player wants to set up a jump command each time they open their mouth, Enabled Play will identify that person’s individual resting mouth position and set it as a reference.

In January, Enabled Play was officially launched in six countries – its user base spanning from the US to the UK, Ghana and Austria. For Dunn, one of its main goals was to fill a gap in affordability and price compared to other assistive gaming devices.

“There are things like the Xbox Adaptive Controller. There are things like the HORI Flex [for Nintendo Switch]. There are things like Tobii, which does eye tracking and stuff like that. But it still seemed like it wasn’t enough,” he said.

Compared to some devices that are only compatible with one gaming system or computer at a time, Dunn’s AI-enabled controller – priced at $249.99 – supports a mix of inputs and outputs. Speech therapists say that compared to augmentative and alternative communication (AAC) devices, which are medically essential for some people with disabilities, Dunn’s device offers simplicity.

“This is just the beginning,” said Julia Franklin, speech pathologist at Davidson Community School in Davidson, North Carolina. Franklin introduced Enabled Play to students this summer and believes it’s a better alternative to other AAC devices on the market that are often “expensive, bulky and limited” in terms of usability. Many sophisticated AAC systems can range from 6,000 $ to $11,500 for high-tech devices, with low-end eye trackers running in the thousands.A person can also download AAC apps on their mobile devices, which range from $49.99 to $299.99 for the app alone.

“For many people who have physical and cognitive differences, they often exhaust themselves learning a complex AAC system that has limitations,” she said. “The Enabled Play device allows individuals to leverage their already present strengths and movements.”

Netizens applauded Dunn for his work, noting that asking for accessibility shouldn’t equate to asking for “easy mode” – a misconception often cited by critics of making games more accessible.

“That’s how you make the game accessible,” wrote one Reddit user of Enabled Play. “Not by dumbing it down, but by creating mechanical solutions that allow users to have the same experience and accomplish the same feats as [people without disabilities].” Another user who said he regularly works with young cerebral palsy patients speculated that Enabled Play would “literally change their lives”.

Accessibility consultants are building a more inclusive gaming industry behind the scenes

But the device is not limited to the gaming sphere. It is also used in schools to make computer labs more accessible. With the rise of remote work and online learning environments brought about by the pandemic, Jaipreet Virdi, historian, author and professor at the University of Delaware, said the device could serve as a model for “inclusive participation ” in schools.

“If students with disabilities can learn and keep up with the expected academic pace thanks to these [assistive] technologies, so they can graduate with more opportunities than their disabled ancestors ever had,” Virdi said.

In some therapy programs in the United States, specialists use Enabled Play to track facial expressions and gamify treatment sessions. Alissa McFall, a speech pathologist and orofacial myologist in Sacramento, said it can be used to analyze how a patient’s muscles are working so healthcare professionals can then use that information to develop personalized treatment plans.

“The greatest value we have seen so far using the Enabled Play device is that it can be programmed to read natural communication movements and connect each sound or facial expression to a meaningful function for an individual,” McFall said.

Since launching in January, Enabled Play has partnered with a number of gaming and assistive technology organizations, including Special Effect, Makers Making Change, and most recently Microsoft with its Partner Program. Accessibility Designed for Xbox. Next, Dunn hopes to roll out “virtual devices” soon, which would allow other developers to add Enabled Play inputs to their apps. With these additions, a person can use facial expressions and voice commands in Microsoft Word and Adobe Photoshop without purchasing a separate device.

As developers look for ways to make technology more accessible, Dunn hopes to help drive that change, encouraging others to think way beyond typical keyboard and mouse inputs.

“It’s a very personal mission for me to solve these problems,” he said. “That’s the difference I’m looking for, which is to build devices that change the paradigm of human-computer interaction to one that’s more inclusive.”

Amanda Florian is a journalist based between the United States and Shanghai. It reports on technology, culture and the new Chinese media scene.

Share.

Comments are closed.