Performing Together:Synchronisation and Communication in Music Ensembles

Communicating, coordinating, and synchronizing thoughts and actions with one another is a fundamental faculty of human beings. Ensemble music performance represents a particular challenge to this ability, because movements and sounds have to be synchronized with highest precision while tempo and other expressive parameters vary permanently over time. The bulk of this world`s music is performed by more than one person, resulting in a wealth of possible combinations: a piano duo for four hands, a classical string quartet, a spontaneously improvising Jazz combo, or a symphony orchestra with choir and soloists, just to name a few. Every possible combination yields its own characteristic dynamics of interpersonal communication. From the democratic organization of small ensembles, in which one musician takes the lead at one moment only to be led by others at the next, the range extends to more hierarchical organizations where many have to follow the sounds or the gestures of one (soloist versus accompanists, conductor versus orchestra). This research project investigates interpersonal synchronization in small music ensembles and the role of gestural communication among the ensemble members. The guiding vision is to understand the underlying mechanisms and learning processes of musical synchronization at a level of detail that can be implemented into computational models that operate in real time. Such real-time frameworks will be employed for interactive experimentation with humans: musicians co-perform jointly with computational models of which the behavioral characteristics are controlled and manipulated (reactivity, disposition to follow or lead, attentional focus to particular ensemble members, etc.). Further experimentation focuses on the motion kinematics in communicative gestures of ensemble members (e.g., pick-up head movements) and the evolution of a common musical goal of each musician over multiple sessions of rehearsal. To measure these complex phenomena, hybrid pianos are combined with optical capturing systems to record the individual performance and the movements of the musicians playing together. Data collected from both real-world performances and controlled laboratory experiments involving both student and expert musicians will be subjected to different modeling approaches (such as dynamical systems or machine learning) to understand the dynamics and mechanisms of interpersonal music making. Applications of this research may emerge for educational settings where computational visualization tools could help to enhance the awareness of movement and sound synchrony among musicians, new performance interfaces for computer music and dislocated interaction, or intelligent accompaniment systems that not only react, but act as full musical companions, even understanding visually the gestures of their human partners.

Interpersonal communication and the coordination and synchronization of actions are fundamental human capacities. People use these functions routinely in activities such as shaking hands, driving a car, playing sports, or playing music as part of an ensemble. To coordinate your actions with someone elses, you must be able to predict how the other person is going to behave. Music ensemble performance provides a particularly interesting context for studying prediction and coordination because the synchronization between actions must be so precise. Since music is dynamic and time-varying, ensemble musicians must make predictions about their co-performers behavior as they play, relying primarily on nonverbal cues provided by their co-performers body movements, breathing, and sound. This research project investigated the mechanisms underlying musical synchronization in small ensembles performing both Classical music and Jazz, using a combination of perceptual and performance experiments and computational modeling techniques.A first series of experiments investigated the importance of different modalities (auditory, visual by nonverbal signals, breathing, cueing gestures) musicians utilize to communicate with each other at the beginning and during an ensemble performance. We found that the most important modality is the auditory; visual information becomes important at points of musical ambiguity such as at piece entrances, after fermatas or during tempo changes. Here in the absence of audio information, instrument-specific motor expertise can facilitate accompanists use of visual cues. In another series of experiments, we focused on the movement properties of cueing-in gestures employed by musicians to begin a piece in synchrony. We found support for the hypothesis that peaks in head and hand acceleration trajectories yield specific timing information about the upcoming performance.A fundamental process to stay in time with an external beat is sensori-motor synchronization, which was studied in this project by implementing and evaluating computational timing models into an artificial accompaniment system, able to co-performa with a (human) musician or tracking automatically a substantial corpus of human piano performances.By analyzing the vertical and horizontal timing of Jazz trio performances and databases of dance music, we attempted to extract and evaluate performative aspects related to the perception of groove. Skilled listeners preferred samples of Jazz trio performance with considerably smaller ensemble asynchronies than originally recorded, supporting a hypothesis towards the appreciation of quantized music.