Computational Style Analysis from Audio Recordings

The project aims at investigating the fascinating, but elusive phenomenon of individual artistic music performance style with quantitative, computational methods. In particular, the goal is to discover and characterise significant patterns and regularities in the way great music performers (classical pianists) shape the music through expressive timing, dynamics, articulation, etc., and express their personal style and artistic intentions. The starting point is a unique and unprecedented collection of empirical measurement data: recordings of essentially the complete works for solo piano by Frederic Chopin, made by a world-class pianist (Nikita Magaloff) on the Bösendorfer computer-controlled SE290 grand piano. This huge data set, which comprises hundreds of thousands of played notes, gives precise information about how each note was played, including precise onset time, duration, and loudness. State-of-the-art methods of intelligent data analysis and automatic pattern discovery will be applied to these data in order to derive quantitative and predictive models of various aspects of performance, such as expressive timing, dynamic shaping, articulation, etc. This will give new insights into the performance strategies applied by an accomplished concert pianist over a large corpus of music. Moreover, by automatically matching these precisely measured performances against sound recordings by a large number of famous concert pianists, comparative studies into individual style differences and commonalities will be performed, which, for the first time, will permit truly quantitative statements about individual artistic performance style. All this requires extensive basic research into new, effective and efficient computational methods for intelligent audio analysis (e.g., extraction of expressive parameters from audio, and precise alignment of different sound recordings) and intelligent data analysis and modelling (e.g., sequential pattern discovery, hierarchical probabilistic models, etc.). The resulting methods will be very general and will have a wide applicability beyond this particular project. The project hopes to make new contributions both to modern empirical musicology - through the discovery and precise characterisation of detailed aspects of expressive performance style - and to Artificial Intelligence and intelligent data analysis - through new methods and algorithms for automatic pattern discovery and machine learning. In addition, it will contribute to international research in these areas by making data and methods available to other scientists around the world. The project can be seen as a logical continuation and extension of work started in a previous project ("Artificial Intelligence Models of Musical Expression"), which was supported by a START Research Prize and in which we managed to show that expressive music performance is indeed amenable to computational modelling and analysis. The current project transcends that initial work by focusing on a very elusive question that is of central interest not only to musicology and the arts world, but also to the general public, namely, the art of great artists, and what makes great music artists unique.

The project aimed at investigating the fascinating, but elusive phenomenon of artistic music performance style with computer methods. In particular, the goal was to study and quantify the way great music performers (classical pianists) shape the music through expressive timing, dynamics, articulation, etc., and express their personal style and artistic intentions. The starting point for the project was a unique and unprecedented collection of empirical measurement data: recordings of the complete works for solo piano by Frederic Chopin, made by a world-class pianist (Nikita Magaloff) on the Bösendorfer computer-controlled SE290 grand piano. This huge data set, which comprises measurements of hundreds of thousands of played notes, gives precise information about how each note was played, including precise onset time, duration, and loudness. In laborious work, this data set was prepared for computational analysis (by encoding the scores of Chopin`s complete piano works in computer-readable form, relating each played note to the corresponding written note, identifying and classifying each error made by the pianist (there were some 30.000 of these!), etc.). New computer algorithms needed to be developed to solve this task. In order to be able to analyse such a huge corpus of measurement data, new computer methods for visualising expressive timing and dynamics were developed (based on the notion of phase planes as used to describe dynamical physical systems). Various aspects of Magaloff`s playing style and technique were studied and quantified - e g., we performed a detailed analysis of the pianist`s errors and mistakes, and of his way of using asychronies between the two hands to generate expressive effects. Also, various specific hypotheses by musicologists were empirically evaluated (e.g., theories that relate expressive timing to models of human motion, and theories of how pianists cope with old age in performance). Machine learning algorithms were used to search for, and learn, predictable patterns in Magaloff`s playing - general strategies he seems to use to make his Chopin renditions sound musical and expressive. One result of this was a computer model of expressive piano performance, named YQX, which predicts the most likely or plausible way in which a pianist would play a given musical passage (in terms of tempo and loudness changes etc.). We submitted this computer model to an international scientific "computer piano performance contest" (RENCON 2008, Sapporo, Japan), where it had to play two entirely new (unknown) piano pieces on a computer grand piano; YQX won all three main RENCON prizes with its interpretations. To summarise, the main project results are: (1) the "Magaloff Corpus", a resource of performance measurement data that is unique worldwide in terms of size, complexity, and artistic quality; (2) a predictive model (YQX) of expressive piano performance that seems to capture musically relevant principles of piano playing; (3) new insights of interest to musicology (published in musicological journals); (4) new computational methods for performance visualisation, analysis, learning, and prediction; (5) (not mentioned above) computer algorithms that can follow and track expressive music performances in real time and synchronise a performance with arbitrary other events. And last, but by no means least: the Wittgenstein Prize awarded to project leader Gerhard Widmer in 2009 can also be seen as a consequence of the long-term support of this kind of research by the FWF.