On Valid and Reliable Experiments in Music IR

Every experimental science is based on the notion of valid and reliable experiments, i.e. experiments that really measure what one wants to examine and experiments which yield repeatable results. Music Information Retrieval (MIR), as the interdisciplinary science of retrieving information from music, conducts experiments with a multitude of methods from machine learning, statistics, signal processing, artificial intelligence, etc. It relies on the proper evaluation of all these methods to measure the success of new algorithms, or, in more general terms, chart the progress of the whole field of MIR. The principal role of computer experiments and their statistical evaluation within MIR is now widely accepted and understood, but the more fundamental notions of validity and reliability in MIR experiments are still rarely discussed within the field. This lack of awareness for valid and reliable MIR experimentation is at the heart of a number of seemingly puzzling phenomena in recent MIR research. Marginally and imperceptibly altered data, so-called adversarial examples, are able to drastically reduce performance of state of the art MIR systems. It has even been claimed that such easily fooled MIR systems therefore do not use musical knowledge at all. Other authors have pointed out that, due to a lack of inter-rater agreement when annotating ground truth data, performance in many MIR tasks can never exceed a certain glass ceiling, since it is not meaningful for an algorithm to model specific raters. A problem of algorithmic bias are difficulties of learning in high dimensional spaces, where some data objects act as `hubs`, being abnormally close to many other data objects thereby causing disturbances in music recommendation, since hub songs are being recommended over and over again. Although a small but growing body of work and literature concerning these MIR problems exists, what is still lacking is an understanding of their true nature: they are problems of validity and reliability in MIR experimentation. Since a failure to comprehend this fundamental issue at the heart of MIR is severely impeding progress in the field, our main goals in this project are: (i) to provide a framework for valid and reliable experimentation in MIR; (ii) to advance the state of the art concerning adversarial examples, inter-rater agreement and algorithmic bias by conducting exemplary valid and reliable MIR experiments. The main focus of this project is on MIR where the above mentioned phenomena are especially apparent, but the very same problems of course have ramifications in general machine learning also, making sure that our research has the potential to advance the progress in MIR and far beyond.

On Valid and Reliable Experiments in Music Information Retrieval (MIR) Every experimental science is based on the notion of valid and reliable experiments. Validity is the truth of an inference made from evidence, such as data collected in an experiment, while reliable experiments are experiments which yield repeatable results. MIR, as the interdisciplinary science of retrieving information from music, conducts experiments with a multitude of methods from machine learning, statistics, signal processing, artificial intelligence, etc. It relies on the proper evaluation of all these methods to measure the success of new algorithms, or, in more general terms, chart the progress of the whole field of MIR. At the outset of this project, the principal role of computer experiments within MIR was already widely accepted and understood, but the more fundamental notions of validity and reliability in MIR experiments were still in need of thorough discussion and clarification. This was clearly apparent when we researched a number of seemingly puzzling phenomena in MIR research and understood their true nature - they are problems of validity and reliability: (i) marginally and imperceptibly altered data, so-called adversarial examples, are able to drastically reduce performance of state of the art MIR systems (lack of construct validity and reliability); (ii) due to low inter-rater agreement when annotating ground truth training data for MIR systems, performance in many MIR tasks can never exceed a certain glass ceiling, since perfect performance can only be achieved for individual annotators, never for a group of users that are in disagreement (lack of external validity and reliability); (iii) a prominent problem of algorithmic bias are difficulties of learning in high dimensional spaces, where some data objects act as "hubs", being abnormally close to many other data objects thereby causing unfair music recommendation, since hub songs are being recommended over and over again (lack of internal validity). In our project we were able to advance the state of the art concerning adversarial examples, inter-rater agreement and algorithmic bias by conducting exemplary valid and reliable MIR experiments. Most importantly our main result is a report and theoretical framework discussing what a valid and reliable experiment in MIR is. To achieve this, we illustrated four major types of validity and discussed threats to each type arising during experiments. Our discussion was grounded with a prototypical MIR experiment on music classification. We also provided concrete guidance to MIR practitioners on how to make valid inferences from data collected from their experiments. All this together aims to bring within the realm of MIR what validity means, why it is important, and how it can be threatened.