The next generation of hearing aids could read lips through masks

Conceptual illustration of the proposed lipreading framework. The framework uses Wi-Fi and radar technologies as enablers of RF sensing-based lip reading. A dataset including vowels A, E, I, O, U and empty (static/closed lips) is collected using both technologies, with a face mask. The collected data is used to train ML and DL models. Credit: Nature Communication (2022). DOI: 10.1038/41467-022-32231-1.

A new system that can read lips with remarkable accuracy even when speakers are wearing face masks could help create a new generation of hearing aids.

An international team of engineers and computer scientists developed the technology, which for the first time combines radio frequency sensing with artificial intelligence to identify lip movements.

The system, when integrated with conventional hearing aid technology, could help combat the “cocktail effect”, a common shortcoming of traditional hearing aids.

Currently, hearing aids help people who are hard of hearing by amplifying all the ambient sounds around them, which can be helpful in many aspects of daily life.

However, in noisy situations such as cocktail parties, the wide amplification spectrum of hearing aids can make it difficult for users to focus on specific sounds, such as a conversation with a particular person.

A potential solution to the cocktail effect is to manufacture “smart” hearing aids, which combine conventional audio amplification with a second device to collect additional data for improved performance.

While other researchers have had success using cameras to facilitate lip reading, collecting video footage of people without their explicit consent raises concerns for individuals’ privacy. Cameras are also unable to read lips through masks, a daily challenge for people who wear face coverings for cultural or religious purposes and a wider problem in the age of COVID-19.

In a new article published today in the journal Nature Communication, the University of Glasgow-led team describe how they set out to harness cutting-edge sensing technology to read lips. Their system maintains privacy by collecting only radio frequency data, with no accompanying video footage.

To develop the system, the researchers asked male and female volunteers to repeat the five vowels (A, E, I, O, and U) first without a mask and then while wearing a surgical mask.

As the volunteers repeated the vowel sounds, their faces were scanned using radio frequency signals from both a dedicated radar sensor and a wifi transmitter. Their faces were also scanned while their lips remained still.

Then, the 3,600 data samples collected during the scans were used to “teach” machine learning and deep learning algorithms how to recognize the characteristic lip and mouth movements associated with each vowel sound.

Since radio frequency signals can easily pass through the masks of volunteers, the algorithms could also learn to read the vowel formation of masked users.

The system was found to be able to read volunteers’ lips correctly most of the time. Wifi data was correctly interpreted by the learning algorithms up to 95% of the time for unmasked lips, and 80% for masked ones. During this time, radar data was interpreted correctly up to 91% without a mask and 83% of the time with a mask.

Dr Qammer Abbasi, James Watt School of Engineering, University of Glasgow, is the lead author of the paper. He said: “Around 5% of the world’s population, or approximately 430 million people, suffer from some kind of hearing loss.

“Hearing aids have brought transformative benefits to many people with hearing loss. A new generation of technology that collects a wide range of data to increase and improve sound amplification could be another major step in improving the quality of life for people with hearing loss.

“Through this research, we have shown that radio frequency signals can be used to accurately read vowel sounds on people’s lips, even when their mouth is covered. While lipreading results with radar signals are slightly more accurate, Wi-Fi signals also demonstrated impressive accuracy.

“Given the ubiquity and affordability of Wi-Fi technologies, the results are very encouraging, suggesting that this technique has value both as a stand-alone technology and as a component of future aids. multimodal auditory.”

Professor Muhammad Imran, Head of the Communications, Sensing and Imaging Research Group at the University of Glasgow and co-author of the paper, added: “This technology is the result of two research projects funded by the Engineering and Physical Sciences Research Council (EPSRC), called COG-MHEAR and QUEST.

“Both are aimed at finding new ways to create the next generation of healthcare devices, and this development will play a major role in achieving that goal.”

The team’s paper, titled “Pushing the Limits of Remote RF Sensing by Reading Lips Under the Face Mask”, is published in Nature Communication.

How to know if you need an over-the-counter hearing aid or a doctor

More information:

Hira Hameed et al, Pushing the Limits of Remote RF Sensing by Reading Lips Under the Face Mask, Nature Communication (2022). DOI: 10.1038/41467-022-32231-1.

Provided by
University of Glasgow

The next generation of hearing aids could read lips through masks (2022, September 7)
retrieved September 7, 2022

This document is subject to copyright. Apart from any fair use for the purpose of private study or research, no
any part may be reproduced without written permission. The content is provided for information only.

Stay connected with us on social media platform for instant update, click here to join our Jwitter& Facebook

Source link