Social Media Mining for Multimodal Music Retrieval

Online social media has been seeing an incredible boom during the last couple of years. Even though they are researched heavily by the Data Mining and Information Retrieval communities, mining, analyzing, and exploiting the user-generated data offered on various platforms such as Twitter, Facebook, or Youtube, is still underrepresented in the context of Music Information Retrieval (MIR). The sole exception to this lack of exploitation is the music service Last.fm, which is used to a certain extent in scientific work by the MIR community, albeit only for a limited number of tasks. Hence, in this project we propose to comprehensively research the use of a large number of different user- generated data sources for several innovative tasks in MIR research. To this end, we identified three goals that will be thoroughly addressed: 1. Inferring Geospatial Listening Patterns from User-generated Data 2. Multimodal Modeling for Serendipitous Music Access 3. Building Intelligent, Adaptive User Interfaces to Music Collections In the first challenge we will investigate whether social media can be used as reliable source to derive music listening patterns on a worldwide perspective. Upon identification of such patterns, we will be able to study geospatial aspects of music consumption and in turn look into methods to predict the popularity and trendiness of music items by fusing data from different sources. We will further research methods to detect novelty (e.g., novel releases, up-and-coming artists). In the second challenge, we will develop multimodal approaches to model the user and the music items. Defining and building user models from his or her digital traces in social media will be enabled by machine learning techniques dedicated to user-generated data. Multimodal representations of music items (such as video clips, song lyrics, cover artwork and audio files) will be generated by elaborating novel feature extractors and fusion techniques. Integrating multimodal descriptors of users and music items, we will then research ways to create serendipitous music access models and systems in the third challenge. Incorporating these different modalities will enable to develop personalized retrieval schemes, paying particular attention to the aspects of similarity, diversification, novelty, and familiarity/popularity. Based on the elaborated retrieval models, we will eventually create prototypical intelligent user interfaces to explore music collections. Expected results of the realization of the project include: deeper understanding and exploitation of social media for music-related activities novel user models dedicated to the music domain serendipitous music retrieval and recommendation systems novel multimodal models of music items highly efficient algorithms to identify music items in user-generated data streams understanding the emergence and evolution of music trends extending research in user-centric MIR intelligent user interfaces to explore music collections

The project had three goals, i.e., (1) inferring geographic listening patterns from user- generated data, (2) multimodal modeling for serendipitous music access, and (3) building intelligent, adaptive user interfaces to music collections. Following a highly user-centric perspective and a data-driven research approach, we exploited user-generated data and conducted detailed surveys in order to uncover relationships between listeners emotions, personality traits, demographics, cultural background, and location, as well as music perception and preferences towards music consumption (Goals 1 and 2). We then formalized the results in listener models, further addressing aspects of diversity, mainstreaminess, and novelty of music taste (Goal 2). Integrating these models into music recommender systems algorithms yielded substantial improvements of recommendations (Goal 3). In addition, we created a prototype of an adaptive music listening interface to explore music listened to in all parts of the world (Goal 3). In terms of basic research, we would like to highlight the following major results and findings: Acquisition, establishment, and decent analysis of a large, standardized dataset of more than one billion music listening events together with listener-specific data (demographics and newly designed features such as mainstreaminess or novelty). Studies on country-specific music preferences uncovered large differences between countries, with some countries largely following a global mainstream (e.g., UK or the Netherlands), whereas others establishing strong local tastes (e.g., Japan or Finland). We showed that these differences can be related to cultural and socio-economic factors on the country level. On a listener level, we showed that correlations between personality traits and music preferences exist. Using machine learning, we were able to predict listeners and countries music tastes from demographic and from cultural information, respectively. We developed an algorithm to infer personality traits by fusing information from various social media. We devised an approach to automatic music playlist continuation based on deep learning and a fusion of audio and textual features. To conclude, we would like to mention that the project also attracted considerable public interest, evidenced by various articles in distinguished newspapers and journals for popular science communication, among others, Der Standard and Salzburger Nachrichten. The project was also selected by FWF as Project of the Week in December 2016.