Rach3: A Computational Approach to Study Piano Rehearsals

In Western classical music, expressivity is crucial to our enjoyment of music, motivating us to attend live concerts and seek out different interpretations of the same pieces. While most research on expressive music performance focuses on the final product (i.e., a live performance in front of an audience, or a recording of a performance in studio/lab settings), musicians actually spend most of their time practicing rather than performing in front of an audience. Music rehearsal is an extended process, taking place over months or years, and involves skill development, evolving interpretations, and occasional periods of decline followed by re-learning. To understand the rehearsal process, a comprehensive zoomed-out perspective is required, as well as the capability to zoom-in to examine individual rehearsal sessions. But practice as a long-term process is difficult to capture: getting people to come and practice for 15 minutes in the lab is not sufficient, and just capturing audio or video is also insufficient. This project presents a computational, data-driven approach to study music rehearsal over long periods of time, leveraging current advancements in artificial intelligence and machine learning. The project focuses on piano music, as it is not only one of the most popular instruments but also among the most extensively studied in music research. The central questions of this project concern the strategies musicians employ during practice and the evaluation of performance quality in relation to the final product throughout the rehearsal phase. A central contribution of this project is the Rach3 dataset, a large-scale dataset containing audio, video, and MIDI data, enabling computational analysis of pianists musical sound and body motion. A large part of this dataset focuses on the PI`s learning of Rachmaninoff`s challenging 3rd Piano Concerto (and hence the name of the project). This project bridges music cognition, performance science, computational musicology, and computer science. From a music cognition standpoint, the project allows for real-world study of long-term practice, using a comprehensive approach that includes aspects like motor control and movement patterns. From a performance science perspective, the project investigates the essential skills and mechanisms that underlie the rehearsal process and the experiences it entails. The computational musicology component of the project enables the testing of hypotheses regarding the development of musical interpretation and comprehension of structural elements within a piece. Lastly, the computational aspect of this project presents an interesting case for expanding tools from Music Information Retrieval for analyzing complex musical data.