Expression, Emotion and Imagery in Music Performance

In FWF Lise Meitner Project M 1186-N23 (2009-2011), we are studying how performed accents (timing, dynamics, articulation) depend on immanent accents (structural, metrical, harmonic, melodic). By adapting Director Musices (DM), a system of context-dependent performance rules (Sundberg et al., 2003), we have achieved a new balance between the local (individual accents) and the global (hierarchical phrase structure) (Bisesi & Parncutt, 2011; Bisesi et al., 2011; cf. Cook, 2011). Like accents, emotions may be performed (added during interpretation) or immanent (latent in the score, e.g. a meditative nocturne, a heroic march). We now ask: Which emotions are elicited by what kind of performed emphases at which immanent accents? What connects the timing/dynamic shaping of individual accents to emotions or implied motion? What are the consequences for musical meaning (imagery, associations, metaphors, archetypes)? In musicians` descriptions of performances, how are their words cognitively organised? How might well-founded answers to such questions be integrated into current interfaces for expressive control of music performance? In addressing these issues we will take advantage of rules for structural, emotional and gestural expression (GERM, Juslin et al., 2002) and multidimensional representations of (musical) emotions (Russell, 1980; Schubert, 1999). Sloboda (1991) relates immanent structural elements such as sequences and unexpected harmonies to physiological responses like shivers and tears, anticipating our local, accent-based approach; we will explore to what extent a reductionist approach can predict the emotional quality of a musical passage. Subprojects include: (1) extension of the MIR toolbox (Lartillot & Toiviainen, 2007) to include immanent accents as a property for automatic style classification; (2) extension of Friberg`s pure data-DM for real-time expressive control of music performance by accent-oriented algorithmic formulations and construction of gestural interfaces based on new qualitative data; (3) algorithmic generation of interpretations of piano works that differ in the selection of events for emphasis, the kind of emphasis, and the intended emotion in order to be aesthetically evaluted; and (4) development of a procedure for the extraction of timing and dynamic profiles from audio recordings in order to classify performers` styles and map performances into multidimensional spaces of emotions. Various empirical research methods will be implemented, including grounded theory, multidimensional scaling, and correspondence analysis. Collaborators will include Richard Parncutt (Graz, music psychology), Anders Friberg (Stockholm, performance rendering), Petri Toiviainen (Jyväskylä, emotion models), Sergio Canazza (Padova, sound and music computing) and Giuseppe Cabras (Udine, digital signal processing).

Music expression is fundamental for music performance. It involves communication of emotion and meaning to listeners. Performers make their performances uniquely expressive by intuitively analysing the musical structure and on that basis trying to understand and implement the composers sonic intentions. They then creatively add their original, personal interpretations. In practice, they do this by manipulating timing and dynamics in response to features of the musical structure (e.g., phrasing, accents, motives, form). This influences the performances gestural character, and the imagery and meaning that it communicates to listeners.The current project was a follow-up to FWF Lise Meitner Project M 1186-N23, whose aim was to explore the relationship between music structure and expression. In this stand-alone project (P 24336-G21), results of previous studies were implemented in computer models, allowing us to address new questions about the relationship between musical structure and emotional communication. We (i) tested and improved existing models of structurally based (immanent) accent salience in tonal music; (ii) developed a procedure for the automatic analysis of musical structure and expression in audio recordings; (iii) used this software to classify performers styles and map performances into a multidimensional spaces of emotions; and (iv) explored the relationship among structure, expression and meaning including affective / sensorial categories and intuitive physical metaphors.These points were addressed in several subprojects, including: (1) extension of Project M 1186-N23 to incorporate a style-dependent model for harmonic accents / tension, that was then empirically tested; (2) relationship between musical structure, as predicted by accent theory, and perceived emotion in a specific repertoire (variations on the Theme of La Folia by different composers); (3) a qualitative study on the informal vocabulary for musical structure, expression, emotions and associations in piano music, involving listeners with different degrees of expertise; (4) development of new methods in music information retrieval for measuring timing and dynamics in expressive piano performance; (5) classification of music performances according to performers emphasis on different kinds of accent and their individual strategies for communication of structure, expression and emotion; (6) application of new algorithms and methods to investigate historical compositional trajectories (from late Schubert to late Liszt, Wagner, and early Schoenberg) from the double perspective of music analysis and music psychology.Our project contributed to music performance research by bringing together mainstream approaches to music psychology and music analysis. In addition, we considered performers intentions and the means by which they control their performances. Our methods combined top-down and bottom-up approaches as well as qualitative and quantitative techniques. We departed from the relatively reductionist approach of previous psychological research on music expression, adopting a more complex model that also considers the knowledge and intuitions of music performers and music analysts.Several international collaborations were included: KTH Stockholm, CSI Lugano and universities in Jyväskylä, Padova, Udine, Bologna, and Montreal.