Adaptive Audio-Visual Dialect Speech Synthesis

The goal of this project is to investigate multimodal adaptation for audio-visual speech synthesis. Human speech is multimodal and therefore we aim at modeling both the audio and visual signals jointly. Furthermore, in speech behavior we are confronted with intra-speaker variability (e.g. variability in dependence on different speech situations, speaking tasks or emotional states of the speaker) and inter-speaker variability (e.g. variability across sociolects and/or dialects). The second type of variation can be modeled by adapting average models of speakers with different dialects to a speaker of a specific dialect. Dialect is chosen as a source of variation between speakers to extend our previous work on Viennese sociolects to other Austrian dialects and to conduct basic research on the audio-visual synthesis of dialects. Generally, audio-visual speech synthesis is the attempt to generate speech and visual signals of a person speaking. In most previous approaches the acoustic and visual signal were modeled separately although both signals are the result of the same underlying articulation process and should be treated as one. Moreover, adding visual information to the synthesis models might lead to better overall acoustic synthesis. Therefore, we propose a joint audio visual modeling framework that is able to generate both acoustic and visual speech for different Austrian dialects. By employing hidden Markov models (HMMs) for both audio and visual speech synthesis, it is possible to combine these two feature streams into a single model. Therefore, a major aspect of this project will be the multimodal adaptation of audio-visual synthesis models. The joint adaptation of audio and visual models from multimodal audio-visual models has not yet been investigated and leads to several important research questions that we want to address in this project.

Generally, audio-visual speech synthesis is the attempt to generate speech and visual signals of a person speaking. Audio-visual synthesis can be used in communication technologies and computer games. In this project we investigated multimodal modelling for audio-visual dialect speech synthesis. Human speech is multimodal and therefore we modeled both the audio and visual signals jointly. In most previous approaches the acoustic and visual signal were modeled separately although both signals are the result of the same underlying articulation process and should be treated as one.In this project we could show that joint modeling of visual and acoustic signals can lead to better visual synthesis, without changing the quality of the acoustic synthesis. Through the use of flexible models that can be adapted through parameters, these models can be easily reused and transformed.Furthermore we could show that adaptation of visual average models with new data can improve modeling compared to models that dont use any background data. With this method it is possible to train a visual model of a person with a small amount of adaptation data.For controlling acoustic models with a large number of parameters, we developed a method that allows for changing acoustic parameters via visual parameters. The opening of the mouth in the visual model can thus lead to the corresponding acoustic changes in the acoustic model.For modeling of dialects we did extensive recordings of two Austrian dialects with 8 speakers, one Middle Bavarian dialect from Upper Austria (Bad Goisern) and one South Bavarian dialect from Tyrol (Innervillgraten). For these audio-visual dialect recordings for speech synthesis, we developed a method for phonetic data collection and audio-visual recording. For modeling of dialects we developed methods for optimal use of dialect data. The recorded data is already used in other running projects and will also lead to new findings in the future.