Intuitive Collaboration with Household Robots inEverydaySett

In recent years, there has been a rapid increase in the design and development of social service robots, which are intended for being deployed for menial tasks. Such robots need to be designed to interact with people who have little or no prior experience in interacting with a robot. Naive users need to be able to control and collaborate with these robots. Therefore, further research is needed to address the needs of nonexpert users through intuitive methods of interaction. To fill this niche, we will develop methods to better understand human-robot interaction through a collaborative robotic interface for humanoid domestic service robots, where the user can interact with the robot intuitively using multimodal cues of language, gestures and gaze. The project will go beyond the state of the art by combining vision- and gaze-based modalities with language to understand the users intentions for successful task completion. We will use a user-centric approach by conducting user studies to model how a human uses gestures and gaze naturally to collaborate with a robot. These studies will be conducted for scenarios including different tables and shelves as well as tidying up an entire room to show the generality of the approach. The scenarios will be developed both in Virtual Reality (VR) and real-world environments. The VR settings will be simplified replicas of the real-world scenarios and allow for quick prototyping and initial data collection with naive users in the design phase of the project. We will then incorporate these models into a multimodal interface for human-robot interaction. For this, we will develop a graph-based representation of visual scenes that incorporate gaze information, and from which gestural information can be extracted. The novel language module will detect the presence of ambiguous speech parts to augment them with gaze, gesture, or visual information from graph-based representations. The project aims at a completely new approach on how to interact with robots and other cyber-physical systems. The idea is that a novice user can interact with a robot via natural language with intuitive gestures and gaze, as compared to previous collaborative systems that use program-like language commands. The interface will be evaluated by conducting user studies in a real-world environment, for both sets of scenarios, to study the facilitatory and inhibitory effects of employing multiple modalities in communication. We will test the hypotheses that incorporating gesture- and gaze-based information in human-robot interaction makes the human user more comfortable and more trusting towards the robot. The WEAVE project is led by Bipin Indurkhya from University of Cracow, with Michal Vavrecka, CTU Praha, and Markus Vincze from TU Wien.