Towards automatic annotation of electro-acustic music

This project is about automation of annotation of electro-acoustic music through application of machine learning methods. Electro-acoustic music is made with electronic technology, using synthesised sounds or prerecorded sounds from nature and studio which are often extensively processed and altered. Compared to the analysis of instrumental or vocal music, annotation of electro-acoustic music is both more challenging and less developed. There exist no "pre-segmented" discrete units like notes, there is no score and no universally established system for analysis. Although musicology has developed various sets of tools for analysis of electro-acoustic music, the tediousness of manual annotation has prevented the application of these theories to a larger body of music. On the other hand, Music Information Retrieval has developed a rich repertoire of machine learning algorithms for analysis of music, including methods that can be used for automatic annotation. This project is about bringing together musicological theories of electro-acoustic music and machine learning methods with the aim of taking a great step forward towards automation of annotation. Automatic or semi-automatic annotation will considerably accelerate the process of annotation and objectify and hence stabilise the analysis` results. This will also allow generation of reproducible annotations following established, stable and quantifiable standards. This will in turn significantly advance the theoretical coverage and development of electro-acoustic music and bridge the gap between existing musicological theories of electro-acoustic music and the scientific practice. We will build upon an existing framework of analysis of electro-acoustic music: the theory of Spectromorphology developed by British composer Denis Smalley. This theory is concerned with identifying "carriers of meaning", i.e. structural and sonic entities in electro-acoustic music. It covers aspects from very low-level sound properties to highly abstract concepts of form and interdependencies of musical matter. Our project will answer the following research questions: - How can machine learning be employed to identify carriers of meaning within works of electro-acoustic music? - Building upon the theory of Spectromorphology, which level of abstraction/complexity can be reached? - Which parts of the analysis have to be done manually, which parts can be fully automatic?

This project was about automation of annotation of electro-acoustic music through application of machine learning methods. Electro-acoustic music is a contemporary form of electronic composition and production of music that originated in the 1940s. It is made with electronic technology, using synthesized sounds or prerecorded sounds from nature and studio which are often extensively processed and altered. Compared to the analysis of instrumental or vocal music, annotation of electro-acoustic music is both more challenging and less developed. There exist no "pre-segmented" discrete units like notes, there is no score and no universally established system for analysis. Although musicology has developed various sets of tools for analysis of electro-acoustic music, the tediousness of manual annotation has prevented the application of these theories to a larger body of music. On the other hand, Music Information Retrieval has developed a rich repertoire of machine learning algorithms for analysis of music, including methods that can be used for automatic annotation. Machine learning is a subfield of artificial intelligence and is concerned with the design and development of algorithms and techniques that allow computers to learn from data (e.g. the relationship between audio representations of music and semantic descriptions). Our essential result is that machine learning methods can indeed be used for annotation of electro-acoustic music but only in an interactive setting. Only the integration of a human analyst into the workflow allows to sidestep the seeming impasse that the lack of a `ground truth` in annotation of electro-acoustic music presents. Already annotations of traditional music are communal, cultural constructs in their social context rather than objective `ground truths`. This is even more the case for electro-acoustic music with its inquisitive nature and constant exploration and deconstruction of established musical parameters at its very heart. A human analyst in front of the computer, taking all the analytical decisions and also interpreting the output of a repertoire of machine learning algorithms is able to compensate for the lack of semantic comprehension on the side of the computer. In our project we developed two approaches to such interactive exploration into the structural and sonic nature of electro-acoustic compositions. One provides a structural overview of complete pieces of music while the other allows identification and clustering of representative sound groupings at a more detailed level. To show the potential of our methods, we also applied our interactive approaches to two renowned compositions of electro-acoustic music, namely John Chowning`s "Turenas" and Denis Smalley`s "Wind Chimes". This bringing together of musicological theories of electro-acoustic music and machine learning methods will help to accelerate the process of annotation as well as stabilize the analysis` results and make them more reproducible. This will in turn contribute to the theoretical coverage and practice of electro-acoustic music.