Vision-based Human-Robot Cooperation: Perspective Sharing

Imagine a human and a robot performing a joint assembly task such as assembling a shelf. This task clearly requires a number of sequential or parallel actions and visual information about the context to be negotiated and performed in coordination by the partners. However, human vision and machine vision differ. On one hand, computer vision systems are precise, achieve repeatable results, and are able to perceive wavelengths invisible to humans. On the other hand, sense-making of a picture or scenery can be considered a typical human trait. So how should the cooperation in a vision-based task (e.g. building something together out of Lego bricks) work for a human-robot team, if they do not have a common perception of the world? There is a need for grounding in human-robot cooperation. In order to achieve this we have to combine the strengths of human beings (e.g. problem solving, sense making, and the ability to make decisions) with the strengths of robotics (e.g. omnivisual cameras, consistency of vision measures, and storage of vision data). Therefore, the aim is to explore how human dyads cooperate in vision-based tasks and how they achieve grounding. The findings from human dyad will then be transferred in an adapted manner to human-robot interaction in order to inform the behavior implementation of the robot. Human and machine vision will be bridged by letting the human "see through the robot`s eyes" at identified moments, which could increase the collaboration performance. User studies using the Wizard-of-Oz technique (the robot is not acting autonomously, but is remote-controlled by a "wizard" behind the scenes) will be conducted and assessed in terms of user satisfaction. The results of human-human dyads and human-robot teams will be compared regarding performance and quality criteria (usability, user experience, and social acceptance), in order to gain an understanding of what makes human-robot cooperation perceived as satisfying for the user. As there is evidence in Human-Robot Interaction research that the cultural-background of the participants and the embodiment of the robot can influence the perception and performance of human-robot collaboration, comparison studies in the USA an Japan will be conducted to explore if this holds true for vision-based cooperation tasks in the final stage of the research undertaking. With this approach, it can be systematically explored how grounding in human-robot vision can be achieved. As such, the research proposed in this project is vital for future robotic vision projects where it is expected that robots share an environment and have to jointly perform tasks. The project follows a highly interdisciplinary approach and brings together research aspects from sociology, computer science, cognitive science, and robotics.

Humans are willing to help a robot in need in order to keep cooperation with it alive. In her Hertha-Firnberg project Vision-based Human-Robot Cooperation: Perspective Sharing, Astrid Weiss from the Automation and Control Institute (ACIN) at the Vienna University of Technology explored how the collaboration between humans and robot can be improved. At a first glance, one could wonder why a machine that is supposed to be helpful should actually ask a human for help. Should it not be the other way around? Yes it should. However, it is still a long way to go until service robots can autonomously perform the variety of household chores we expect them to do in an error-free manner. As any other service, robots can also fail and we need to provide recovery strategies for human users in order to keep the cooperation alive. When we have to cooperate with other humans we can put ourselves in their shoes. When working with robots, what is happening on the robot side during cooperation often stays hidden. It is one of the biggest technological challenges for a robot to autonomously recognize that it is stuck in a failure situation and that it can only recover with the help from the user (e.g., the robot is stuck due to an obstacle which needs to be removed so that it can move again). In this project, two studies on human collaboration were performed in which participants were intentionally confronted with unsolvable tasks, in order to gain an understanding of how humans handle such situations. Subsequently, two human-robot collaboration studies were performed in a multi-user set-up, in which a robot was producing pre-scripted failures. The goal was to explore how humans react and the gained empirical data was used to implement an algorithm that enables a robot to autonomously decided which user to ask for help in comparable situations. Finally, as collaboration might differ between cultures we performed a video-based comparison study, exploring if Japanese, US, and Austrian participants react differently to help requests from a robot. Results showed that Japanese participants had a hard time with the concept that the human should help the machine; to their conviction, the machine should run flawlessly or a specialized maintainer should step in. However, US and Austrian participants reacted more as expected: the last user who gave a command or is closest to the robot in a failure situation should help it. To summarize, the research performed in this Hertha-Firnberg project brings us one step closer to having autonomous service robots in domestic environments in near future. Now we know that people are willing to accept that their robot needs a little bit of help from time to time and we can offer mitigation strategies in robotics taking the active participation of a user into account.