Emotion in speech – words are not enough

Does someone’s ability to hear a speech signal give us all the information we need?

Communication entails more than exchanging words. Voices carry vital information about their internal states as well, such as emotion. Listen to the samples below from the Toronto Emotional Speech Test (TESS) – they have the same words but clearly convey different emotions.

🙁
😡
🙂
😮

We can all agree that knowing a talker’s emotional state is important information for a listener. Unfortunately, when it comes to hearing aid benefits, little is known about how hearing aids affect users’ perception of emotion in speech.

I was recently an investigator in a study that looked at how hearing aid use affected the perception of emotion in speech and the recognition of speech spoken with emotion. Interestingly, we found that the use of hearing aids improved listeners’ word recognition performance from 43% correct (unaided) to 68% correct (aided), but that hearing aids did not improve listeners’ ability to identify emotions (38% unaided, compared to 40% aided). In short, hearing aids improved the recognition of what was spoken but hearing aids did not improve the listeners’ ability to identify the emotion being conveyed.

Since this study, I joined several colleagues from Ryerson University and Phonak AG to develop a questionnaire that would allow us to measure hearing handicap related to affective (emotion) speech. The questionnaire we developed, called the Emotional Communication in Hearing Questionnaire (EMO-CHeQ), includes 17 items such as, “I find it difficult to identify the emotions of people speaking on television”.

When we assessed the questionnaire, we found that, on average, adults with hearing impairment experience handicap when listening to emotion in speech and current generation hearing aids do not appear to address this performance deficit. Even more remarkable was that when we looked at people who are very satisfied with their hearing aids, there wasn’t less auditory emotion hearing handicap for this group.

To some extent, this is not too surprising because the focus of hearing instrument research, to date, has more focused on optimizing speech intelligibility and sound quality rather than emotion identification. Importantly, this new research presents us with an important new outcome measure to assess improvements with emotion identification.

If you would like to learn more about our study looking at perception of emotional speech by listeners with hearing aids, read a recently published article in Canadian Acoustics.

Do you like the article?

Comments

3 thoughts on “Emotion in speech – words are not enough”

  1. Interesting article.

    Listening can be a tiring and stressful activity for those with hearing impairment. I wonder to what extent this intense concentration on interpreting the message affects a listener’s ability to derive meaning from those additional cues.

    It raises the question of whether the focus should remain on improving intelligibility and therefore creating a more relaxing listening experience. With that might come better recognition of emotion in the signal?

  2. Great comment and question! Disentangling auditory and cognitive contributions to hearing abilities is a long-standing issue in hearing research (see Pichora-Fuller & Singh, 2006, Trends in Amplification, 10:29 for a review). While our group has not directly investigated your question, the study mentioned in the blog is somewhat related. In that study, aided listening may have been less cognitively taxing than unaided listening. Nevertheless, despite this potential difference in the availability of cognitive resources between the two conditions, emotion-identification performance was unchanged. This suggests that emotion-identification performance can act somewhat independently of considerable improvements to speech-intelligibility performance. Nevertheless, I agree with you (and would be surprised) if the availability of cognitive resources was not, in least in part, a relevant factor regarding emotion identification. Thanks for your comment!

  3. I loved this post! I’ve not considered the emotional information speech carries in relation to hearing aid use and this is clearly very important. Thank you for highlighting this. Particularly for children, speech intelligibility has a major role in hearing aid outcomes for children (for example Dr Susan Scollie’s recent research) but your article has also made me think about the importance of other important cues in communication such as body language and emotional information. These are also important for children to learn to help them communicate as effectively as possible. Thank you for your insights! Jane Woodward

Comments are closed.

Author

Gurjit Singh - Senior Research Audiologist at Phonak
Gurjit Singh

Senior Research Audiologist at Phonak Canada

Gurjit, PhD, is a Senior Research Audiologist at Phonak AG, and Adjunct Professor at Ryerson University and at the University of Toronto. He is interested in factors that contribute to success with audiologic rehabilitation and the consequences of listening to signals that contain or evoke emotional responses.

Articles of interest

Translating clinical research to clinical reality

It takes years for research to evolve to the level of understanding required for clinical application. Dr. Jason Galster, Vice President of Clinical Research Strategy with Sonova, shares insights on the stages of translating research in audiology.

by Jason Galster