Personalized Music Retrieval via Content and Context

Intelligent music retrieval approaches that allow exploring today`s ever growing (private and commercial) music collections are currently based on properties related to the music content or the music context. The music content relates to features extracted from the audio signal, whereas the music context comprises data extracted from external, music-related sources, such as Web pages, Peer-to-Peer networks, playlists, collaborative tags or other user-generated content. External aspects of the music listening situation and environment, which relate to the user context, in contrast, have not yet been explored comprehensively for the purpose of deriving relations between them and certain music aspects. Therefore, this research project aims at thoroughly investigating the user context. Aspects to be analyzed and data to be mined include the situation in which a person listens to music, his or her mood, location, activity, social context, used listening device, and physiological status. We are certain that these factors, and many more user- context-aspects, contribute to the user`s music taste and judgment of music similarity. We are hence going to perform an in-depth analysis of data sources related to the user context, elaborate corresponding feature extractors, employ and develop machine learning techniques to relate music content-, music context-, and user context-features, and build models that aggregate these various aspects of music perception. For features related to music content and music context, we can build upon expertise our research group gained through FWF projects L112-N04 (music content) and L511-N15 (music context). Analyzing the user context, in contrast, is a novel research discipline in the context of the music domain. We are, moreover, well embedded in an international research network and maintain close collaborations with top-researchers working in related fields. In this vein, the project aims at distilling subjective factors that influence music taste and judgment of music similarity and at elaborating holistic music similarity measures. Such similarity measures provide an important theoretical foundation of various music-related applications. They allow gaining a better understanding of the factors that influence whether two artists or pieces of music are rated similar, and are a key part of intelligent music retrieval systems. Based on the elaborated similarity models that reflect the three broad categories of aspects (music content, music context, user context), this project eventually aims at yielding prototypes of a new generation of music retrieval systems, such as automatic playlist generators, personalized recommender systems, or adaptive user interfaces to explore music collections. The methods developed in this project will allow building retrieval systems that take subjective aspects of the user context into account, and may therefore help to transcend the present methods that commonly consider at most two of the three categories.

The project aimed at elaborating methods to personalize music access systems, for instance, digital music players and music recommender systems. To this end, we exploited a variety of data sources. In particular, musical characteristics extracted from the audio signal and information mined from the web and from the users and their surroundings have been acquired, analyzed, and aggregated. User-related aspects are fundamental to elaborate a personalized music player, which represents one of the most important practical project results. In terms of basic research, we would like to highlight the following results and findings: Thorough analyses of various data sources to model the user showed that social media, such as Twitter or Last.fm, as well as sensor and environmental data gathered via smart phones are capable of modeling listening habits. The developed techniques enabled us to create two publicly available datasets of location-specific listening events. These are actively being used for research on context-aware music recommendation and others. We elaborated novel music similarity models based on data mined from the web and from social media, and further improved existing audio-based approaches. Such models are vital to build music recommenders and automatic playlist generation systems. Respective implementations also won several international competitions. Our work on audio signal processing resulted, among others, in novel techniques for rhythm identification in music pieces, which in turn could also yield improvements in music similarity measurement and retrieval. We extracted basic characteristics and listening habits of users from social media and figured out which music recommendation algorithm serves best which user group. We showed that context-based music playlist adaptation on mobile devices is feasible and can reveal interesting relationships between user context and music taste. Furthermore, we were able to predict with significant accuracy which music artist is preferred by a given user in a given context. In addition to various publications in journals and conference proceedings related to the fields covered by the project, we contributed several survey articles and book chapters.To conclude, we would like to mention that the project also attracted considerable public interest, evidenced by various articles in distinguished newspapers and journals for popular science communication, among others, Der Standard and Bild der Wissenschaft.