Artificial Intelligence Models of Musical Expression

START project Y 99 Artificial Intelligence Models of Musical Expression Gerhard WIDMER 19.06.1998 The goal of the project is to develop and use methods from Artificial Intelligence (in particular: Machine Learning) for an in-depth investigation of the phenomenon of expressive music performance. Expressive music performance refers to the phenomenon that musicians `shape` and `structure` pieces of music by continually varying certain musical parameters during playing, based on their interpretation and understanding of the music. That includes dimensions like dynamics (variations of loudness), rubato or expressive timing (tempo deviations), articulation, etc. Expressive interpretation is an important and complex aspect of musical intelligence, and a manifestation of both human creativity and rational, cognitive abilities. The goal is to gain a deeper understanding of this complex domain of human competence and to contribute new methods to the (relatively new) branch of musicology that tries to develop models and theories of musical expression. The focus of the project is on the development and use of methods that are capable of producing new insight into and `knowledge` about the phenomenon. In particular, machine learning and data mining algorithms, if employed properly, have the potential to discover new, hitherto unknown regularities in large amounts of empirical data. Machine learning experiments with real performance data will thus be at the core of the project. More precisely, the main research directions to be pursued include - the formulation, and empirical testing in computer experiments, of explicit models of human music understanding and musical `knowledge`; - the development of learning and data analysis algorithms that can combine such explicitly formulated knowledge with patterns extracted from empirical data; - the implementation of these algorithms in computer programs that analyze given expressive performances and extract (learn) generalized principles that might account for or `explain` the observed data; - the analysis of learned rules and characterizations in the light of musicological theories of expression; - based on that, the development of an integrated theory and operational computer model of the most important aspects of expressive performance. Expected benefits include the discovery of new, hitherto unknown regularities and principles underlying expressive performance, a more detailed understanding of the interplay between musical structure and musical expression, new methodologies of formulating and empirically testing musicological hypotheses and, as a side effect, an array of software tools for musical analysis and hypothesis testing. Influence of the proposed work on the development of the field The proposed research proposed is of an interdisciplinary nature; it concerns both music, research (musicology) and computer science / Artificial Intelligence. I expect benefits of the project for both disciplines involved: For empirical music research (musicology), the main benefits will include an array of new computer-based methods for the formalization and empirical testing of theories concerning musical expression and related phenomena; novel characterizations of musical expression patterns and their relation to aspects of musical structure, discovered by machine learning algorithms; these may then be analyzed from a musictheoretic perspective and might lead to the extension or refinement of existing theories (remark: one of my learning systems has already discovered refinements of expression principles postulated independently by musicologists - see proposal); an integrated model of musical expression that can be simulated on a computer and instantiated with different underlying hypotheses; On the other hand, the project will also deliver results of general usefulness in the areas of machine 1 learning and data analysis. The complex musical application requires the development of new, powerful learning algorithms that can deal will highly structured data and can combine empirical data with prior knowledge in a principled form. In addition, the project will produce practical spin-off results like, e.g., an interactive software environment for music and performance analysis and editing; these might be of interest to institutions in music pedagogy or even commercial users like record companies.