Cross-layer language models for conversational speech

Whereas speech scientists have focused on carefully pronounced speech for a long time, the interest has more and more shifted to language as it occurs in natural conversations. This has two reasons. From a technological point of view, there is an increasing demand for social robots, which in order to become more interactional and social also need to use language naturally. Second, linguists became more interested in natural conversations, as they reveal additional insights to controlled experiments with respect to how speech is processed in our brain. In this project, we aim at improving the automatic recognition of conversational speech, at increasing our knowledge about the human production and perception of conversational speech, and to increase our knowledge and resources for conversational Austrian German. On the basis of conversational speech and chat corpora from German and Austrian speakers, we develop cross-layered language models which include acoustic and semantic contextual information the way humans do. These models will be informed by quantitative phonetic corpus studies and tested in ASR and speech perception experiments. For conducting the linguistic studies, speech technology will be used for creating automatic annotations, acoustic feature extraction and data analysis. Gained linguistic knowledge will then again be incorporated into the language models. This approach requires an interdisciplinary team (engineers and linguists) that works closely together. The PI Dr. Barbara Schuppler (Graz University of Technology) is a young interdisciplinary speech scientist who has shown in two previous FWF projects that her cross-layer principle reaches good results for pronunciation and prosody modelling. The project proposed gives her the opportunity to expand the cross-layer concept to language models, and to establish a research group on conversational speech in Austria. The national partners Prof. Dina El Zarka (Department of Linguistics, University of Graz) and Dr. Roman Kern (Know-Center GmbH) bring long lasting experience to the project. Together, they cover the disciplines speech technology, linguistics, phonetics and natural language processing.

In the last decade, conversational speech has received a lot of attention among speech scientists. On the one hand, accurate automatic speech recognition (ASR) systems are essential for conversational systems and social robots, as these shall become more interactional and social rather than solely transactional. On the other hand, linguists study natural conversations, as these reveal additional insights to controlled experiments with respect to how speech processing works. The works of this project investigate conversational speech to enhance our linguistic knowledge of conversational Austrian German and to use this knowledge to improve ASR systems. For this purpose, the GRASS corpus, a large-scale database of Austrian German conversations, has been annotated with respect to communicative functions annotations suitable for a newly introduced method for quantitative analysis of conversational dynamics. Our work demonstrates that prosodic variation in conversational speech is systematic and linked to semantic and pragmatic context. But how sensitive are ASR systems to prosodic cues and conversational context? Our work suggests that integrating both data-driven and theory-driven components, including linguistic knowledge, can improve ASR, particularly for short utterances. When comparing how ASR systems transcribe conversational speech with how humans transcribe the same utterances, we find that they struggle with the same characteristics of conversational speech (e.g., disfluent sentences, dialectal pronunciation, fast speech rate), but just to a different degree. Finally, the project delivers valuable methods for speech technologists working with low-resource languages and dialects and for working with small datasets of high degrees of variation (e.g., pathological speech, child speech).