AI voice assistant helps the visually impaired learn digital skills

News - 02 October 2024 - Communication

We are glued to our screens. Televisions, computers, tablets, smartphones. They offer a world of possibilities and are – for the most part - easy to use. But what about people who are visually impaired or blind? Are their needs considered when designing these devices? No, says IDE alumnus Konrad Krawczyk. That’s why he created an AI-based voice assistant to help people with visual impairment learn digital skills.

“My father became blind as a result of an accident,” explains Konrad. “As people do, we adapted our lifestyle. For holidays, for example, we would consider which places were accessible. For tech, such as computers or smartphones, however, this was another story. I saw my dad struggling with voice control tools and I knew there must be a better way.”

Voice assistants are simply not well designed, which I still find odd considering the huge companies behind these technologies.
― Konrad Krawczyk

For his graduation project Konrad worked with researchers from the Faculty of Industrial Design Engineering's Inclusive Design Lab, amongst whom included assistant professor Stella Boess. He also contacted Timon van Hasselt, an ICT specialist for visually impaired people, at Koninklijke Visio. Visio is an specialised centre for visually impaired and blind people in the Netherlands. “We had the same vision on inclusive design for this group. We agreed that voice assistants are not well designed, which I still find odd considering the huge companies behind these technologies.”

No end-to-end

Konrad analysed the major flaws in these voice assistants. “The issue with most tools is that voice assistance is never fully end-to-end. There is always a small visual component needed, or you need someone next to you to guide you.”

Here is an example of a bad design: you ask Alexa to play a song and Alexa cannot play it because you don’t have the required app yet. Alexa offers a download link, and says ‘click here to install’, but how do you know where to click? Or how about: you look up something on the web. When the search results appear, the voice assistant says: ‘Here’s what I found’, but it does not read the options out loud. “An alternative solution for these types of situations is for a robotic voice to read everything out loud. However this could be confusing for the user, and there are often no customisation options," notes Konrad.

Large Language Models (LLM) are not always trustworthy. The good news is that this tool is enriched by Visio data, which means the clients can work with relevant information.
― Konrad Krawczyk

So Konrad wanted to create a voice assistant which people can use independently. This assistant needed to be able to adapt to different needs and contexts. For example, due to age differences or people who were born blind versus those who became blind over the course of their lives.

This content is being blocked for you because it contains cookies. Would you like to view this content? By clicking here, you will automatically allow the use of cookies.

AI-powered follow-ups

What better technology to power this tool than Artificial Intelligence? That’s why the IDE alumnus came up with an AI-powered hotline which does follow-up calls with Visio clients who are following tech coaching training. The hotline gives the clients trainings in voice commands over the phone. “The tool uses Large Language Models (LLM), just like ChatGPT, but these LLM are not always trustworthy. The good news is that this tool is enriched by Visio data, which means the clients can work with relevant information.”

“Before turning to voice I considered creating a guidebook in braille as it is tactile and no vision is needed either. But it turned out that not too many people can actually read braille or they do not feel comfortable using it. As I wanted to create a tool that doesn’t require new skills and feels comfortable for everyone, audio turned out to be the best option.”

As I wanted to create a tool that doesn’t require new skills and feels comfortable for everyone, audio turned out to be the best option.
― Konrad Krawczyk

The tool Konrad created is now open source and can be used for anyone who wants to make a contribution to it or use it for themselves.

“After this project I would love to explore my options in the mobility industry. Voice control for self-driving vehicles for example would be a great next step. I think incorporating methods from inclusive design into contexts of mobility could make driving and using public transit more intuitive, or even possible, for millions of people. This could make public infrastructure safer.”