AURA: Augmenting musical interaction via EVAs

In todays world, artificial intelligence (AI) is becoming an increasingly important part of our lives. We are seeing the rise of intelligent AI systems that can interact with humans in meaningful ways, primarily through text and speech. One exciting development in this area is the use of embodied virtual avatars (EVAs)digital, visually humanlike figures that can interact with us through gestures and other nonverbal cues. These avatars could add a profound social and emotional dimension to human-agent interaction. However, creating AI systems that can respond meaningfully to human emotions and intentions, especially through nonverbal communication, remains a major challenge. This project aims to tackle this challenge by bringing together experts in artificial intelligence, musicology, social and cognitive sciences, using piano duet playing as a testing ground. The project will develop two main components 1) an expressive automatic accompaniment system that is able to adapt its performance in real time to accompany a soloist and 2) EVAsthree-dimensional, humanlike holograms, displayed via mixed reality technology, that can play piano alongside a human musician in real time. These two components will be integrated into a system that can intelligibly and emotionally engage in live piano playing with a human pianist. While these EVAs have clear artistic and creative applications, they also provide a unique opportunity for scientific research. We will use the EVA-integrated accompaniment system as a tool for research on social action, cognition and emotion in humans. This project bridges multiple fields: music psychology, by exploring how we express and respond to music; performance science, by studying the dynamics of live musical collaboration; and computational musicology and AI, by creating technologies to scaffold our understanding of expressive interaction.