Computational Music Performance Research, Applied

This project builds on many years of basic research where we investigated various aspects of expressive music performance with novel computational methods, and developed predictive computer models of performance dimensions such as expressive timing, dynamics, and articulation. In the process, we not only made interesting discoveries about this complex art, but also developed a lot of new computational music processing technology (e.g., algorithms for beat and tempo tracking, audio-to-audio synchronisation, performance visualisation, or performance prediction, to name but a few). The goal of the present project is to build on these methods and develop them to the point where they may become efficient and robust enough for real-world applications. Expression in music may seem to be an unlikely topic for application-oriented research, but the fact is that computers that can deal effectively with expressively played music (audio) could make possible entirely new and creative (and commercially relevant) applications. The research to be performed in this project is motivated by three such visionary application scenarios: 1. Multimedia Arts: Real-time visualisation of music-dramatic works on stage (as realised, in particular, by the Ars Electronica Center / Futurelab, Linz). 2. Synthetic Musical "Expression": Naturally expressive synthetic instruments and virtual orchestras. 3. Automatic Feedback on Expressive Performance: Interactive tools for music performance teaching and analysis. From an analysis of the requirements that such applications would pose, we have derived the following three main research goals: 1. Expression Extraction: computer methods for automatically and precisely extracting details of expressive performance (i.e., tempo, timing, dynamics, etc.) from audio recordings, possibly in real time; this also involves methods for the precise alignment of audio recordings to musical scores. 2. Expression Tracking: algorithms for following live performance (audio streams) and aligning them to given scores or reference audio recordings in real time; these methods should also work with incomplete scores or recordings and be extremely robust and reactive in the face of expressive deviations, unforeseen disruptions, errors, etc. A sub-problem here is the development of methods for inducing robust predictive tempo models on-line. 3. Expression Rendering: computational models of specific dimensions of expressive performance that permit the automatic generation of synthetic music performances that sound reasonably musical and "natural"; possibly also methods for interactively modifying or controlling such performances. A number of potential application partners - e.g., the Ars Electronica Center and the Vienna Symphonic Library - are supporting this work (e.g., by supplying test data and audio materials), motivated by the prospect of being able to realise complex application projects later on.

The overall goal of the project was to develop musically intelligent computers that can deal with expressive performances and recordings of classical music in useful ways. The term expressive performance is used here to mean all the things that performing musicians (singers and instrumentalists) add to a piece in order to make it come alive and give the piece a specific character (e.g., the specific choice of tempo, expressive timing (speeding up, slowing down, accentuating notes and chords, changing the loudness, etc.). The motivation of the project was to develop computers that have an understanding of such aspects of music and can support real-world (artistic and music-didactic) applications such as real-time music and performance visualisation; synthetic performers and virtual orchestras; and didactic tools for music performance analysis and teaching. A large number of computational methods were developed from computer programs that can identify the piece being played, simply by listening to a live performance, to computers that listen to live on-stage performances in real time (through a microphone) and are able to track the performers position in the musical piece, regardless of the chosen tempo or whether the musicians make mistakes or leave out entire parts. (This can be used, among other things, for live music visualisation, score display, and to drive a fully autonomous automatic page turning device for musicians). Also, computer programs were developed that learn to play a given musical piece more or less expressively, by shaping tempo and loudness in such a way that the piece sounds lively and (most of the time) natural. These programs won all relevant awards at an international Computer Piano Performance Contest (RENCON 2011, Padova, Italy), where scientists compare their computer models of expressive music performance.Some of the new technologies described above can be seen in action in our YouTube Channel www.youtube.com/user/CPJKU .The project has laid the technological foundations for a number of follow-up projects and cooperations where the technologies will be used in real applications among others, the multi-national European research project PHENICX (http://phenicx.upf.edu), where we cooperate with the world-famous Royal Concertgebouw Orchestra Amsterdam to create new multi-media experiences around classical music concerts.