Coordination and Collaborative Creativity in Music Ensembles

Ensemble musicians must coordinate their playing with high precision to produce a coherent performance. To achieve good coordination, they must be able to predict their co-performers` upcoming actions. Most musicians value originality in their playing, however, and aim to produce unique performances. These conflicting goals raise an interesting question: how do ensemble musicians balance their desire to sound unique with the need to remain predictable to each other? Playing music is an inherently creative task, as it requires the creative interpretation of existing music, if not the creation of new music altogether. This project investigates how ensembles coordinate when playing music that is ambiguous, and therefore places high demands on creative thinking (e.g. improvised music or notated music with a free rhythm). We consider both human-human and human- computer collaborations so that we can test how factors normally present in live human interaction (e.g. the opportunity for communication; the perception of co-performers as in charge of their own actions) affect creative collaboration. Three main questions are addressed: 1) how performers predict each other`s intentions, 2) how performers align their interpretations, and 3) whether musicians perceive computer-controlled partners as less creative or less intentional (i.e. not in charge of their actions) than human partners, causing them to perform differently with computer partners than with human partners. To address these questions, we will run a series of studies in which highly-skilled musicians perform as duos. Two studies will investigate how visual communication contributes to ensemble coordination. For these studies we will use motion capture and eye-tracking techniques, which will allow us to measure performers` movements and simultaneously determine where they are looking. In another study, we will investigate how musicians merge their interpretations, by comparing their solo and duo performances of the same pieces. A fourth study will compare human-human performances with human-computer performances to determine whether people play differently with computers than they play with other musicians, and if so, why. In this final study, a novel accompaniment system recently developed by our group will act as the computer partner for human pianists. This system `listens` to pianists and synchronises its accompaniment with their playing using a timing model, which predicts the timing of upcoming notes based on the timing of previous notes. In the current project, we will extend the accompaniment system so that it can communicate visually through the movements of an avatar head. This project will be the first systematic investigation of how musical creativity is achieved in groups. Guidelines will be generated for systems that enable creative musical collaboration between people and computers, and the upgrades made to the accompaniment system will transform it into a valuable tool for both researchers and music performers.

Ensemble musicians coordinate their playing with remarkable precision. For skilled musicians, precise coordination is possible even before the ensemble has agreed on how they want the music to sound. This project examined ensemble coordination during the early stages of rehearsing notated music, when musicians are exploring different possible interpretations of the score. We examined the role of visual-motor interaction in ensemble coordination. Our primary aims were to assess how much time musicians spend watching each other while playing, show how musicians' movements differ between solo and ensemble performance settings, and explain how different strategies help ensemble musicians to navigate creative musical interaction (e.g., exchanging leader/follower roles or playing more predictably). Furthermore, we aimed to contribute to methodology with a new recording paradigm that integrated optical motion capture with mobile eye tracking, and an expressive accompaniment system that would be capable of playing four-hands piano duets (expressively and adaptively) with a human counterpart. We captured and analyzed recordings of duo performances, and discovered that musicians often looked at each other when playing, particularly during irregularly-timed passages, where there is more expressive freedom. The time musicians spent watching each other increased as they became more familiar with the music and each other. Musicians' body movements were more communicative when they played with a partner than when they played solo. During duo rehearsals, their movements became larger and more coordinated as the session progressed. Musicians' movements were also more communicative when they could see each other than when their view of each other was occluded. The absence of visual contact did not seem to affect the quality of their performance, however, measured in terms of synchronization success. We also found that musicians naturally assumed leader and follower roles when playing together, and that expressive variability (e.g., in timing) was lower when they played together than when they played alone. These findings suggest that musicians change their behaviour in collaborative settings in order to facilitate the predictability of their playing. Putting our findings together, we propose a new hypothesis: rather than supporting basic coordination, visual interaction between ensemble musicians motivates their engagement with each other and the music, allowing them to establish a cognitive resonance that facilitates creative thinking. Our research has contributed to a general shift in focus in the research community. Rather than treating ensemble performance as a simple synchronization task, researchers are focusing on the process of musical creativity, and treating musicians' behaviours as indicators of the creative process. Our new recording paradigm offers a means of capturing these behaviours in great detail. Our accompaniment system, furthermore, offers researchers with a new opportunity to investigate human interaction under controlled conditions.