Audiovisual speech entrainment in deafness

Communication requires listeners to correctly process the acoustic input delivered by a speaker. However, in many life situations the acoustic speech signal is accompanied and preceded by movements of the speaker, that are visually processed by the listener. Especially lip-movements have been shown to be beneficial for understanding speech, particularly in adverse listening situations, e.g. when trying to follow a conversation at a cocktail party. Also, many individuals with hearing loss depend on the ability to lip read. This ability has also been implicated to mediate rehabilitation of speech perception in individuals that become deaf later in development and receive a so-called cochlear implant (CI). This device circumvents the damaged inner-ear by directly stimulating the hearing nerve. However, especially for adults that have been used to "normal" acoustic input, the input via the CI appears very much distorted and the individual has to actively learn to assign meaning to the unusual sound impressions. Unfortunately, clinical outcomes of CI are tainted by extreme variability, leading to frustrating experiences and rejection of the device in some patients. Considering the costs and risks of the intervention, this variability is a serious problem. A common question to all of these raised issues is: How does the processing of lip movements improve auditory processing? We will capitalize on recent developments also in our lab, to track brain regions that faithfully follow speech-related acoustic and visual information at a rate closely resembling the important frequency of syllables. We will test how different levels of acoustic and visual noise impacts this relevant ability of the brain. Importantly, based on ongoing works we predict an involvement of visual brain regions in tracking the acoustic input also when missing in the stimulation. According to our view, this process is relevant when "translating" the visual input (i.e. lip movement) into a "format" that can be used by the auditory system. According to our project, this capability may be highly variable and partially driving the individual`s capability to lip read. Besides of testing normal hearing individuals, we will investigate how this process is changed in deaf individuals, who putatively rely significantly more on visual processing of lip movements. Finally, we will test how this neural tracking process changes following CI and how measures at an early stage could predict rehabilitation success. This project will use innovative approaches to provide important insights on processing of speech- related lip-movements and how these neural processes are affected by auditory deprivation. Our research could break new grounds in understanding how interindividual differences in neural tracking of lip movements (and its associated acoustic signal) could affect CI rehabilitation success, which is an urgent clinical issue.

When speaking, not only is an acoustic signal produced, but there are also accompanying visible lip movements. These lip movements are not a meaningless side effect, but can shape auditory processing in a significant way. This can be seen, for example, in the fact that seeing lip movements helps comprehension in noisy environments. This might be related to brain waves following lip movements. Brain activity also follows the acoustic speech signal, so-called speech tracking, even if the acoustic part of the speech was not presented at all, but only the lip movements without sound. This project investigates both mechanisms (tracking of the lips and the speech signal) with regard to some general details (e.g. in difficult listening situations) and in particular with regard to hearing difficulties up to deafness. For this purpose, we presented distorted speech material in a study and investigated speech tracking across different degrees of distortion (from well understandable to completely unintelligible). For normal-hearing subjects, we found that speech tracking is strongest for moderately distorted speech and decreases for easy-to-understand and incomprehensible speech. This speech distortion is an approximation of the hearing experience with cochlear implants (CI), so in another study we investigated speech tracking in postlingually deafened subjects (i.e. deafness after language acquisition) who received a CI. Recordings of brain activity after 6 and 12 months post implantation show that auditory speech processing normalises over the months, especially in the ear without a CI. In other studies in which the participant groups only saw lip movements, speech tracking was found in a younger participant group while it was reduced in an older group of subjects, which could be related to increasing hearing difficulties in the course of ageing. Prelingually deafened subjects (i.e. deafened before language acquisition) however showed stronger lip tracking. Since this speech tracking is mainly found in brain regions that are responsible for visual processing, we also investigated whether speech tracking is associated with so-called lip reading. We therefore also developed a test for lip reading skills in German, but no effects were found in this respect. This project has stimulated several new research directions focusing on the role of the visual system in language processing.