InSitu - Integrated Situated Visual Scene and Natural Language Understanding for Human Robot Interaction

Recent years have seen major advances in personal and assistive robots (from household robots to robots for elder care). Yet we are still far away from truly natural human-robot interaction in everyday situations and environments. Robust visual scene understanding and natural language understanding on robots are currently two of the major road blocks. We believe that this is in part due to the fact that these two are often treated separately. Note that typical dialogue between humans situated in the same scene is full of cases where vision and dialogue are used jointly to ground a common understanding. Humans tend to look towards an object that they are currently referring to in dialogue, guiding the attention of the dialogue partner. Similarly object attributes extracted from (even partly) parsed utterances like "Could you hand me the red ...`` will guide the search for the respective object. Vice versa, visually observing a scene that is being talked about supports understanding of ambiguous or underspecified utterances while they are being processed - "the red book on the floor`` will most likely refer to a book visible to the speaker and not the one behind her back. So vision and natural language processing can mutually and incrementally constrain each other. In this project, we will tackle the problem of tightly integrating visual scene understanding with natural language understanding. We believe that for robots to reach human-like performance in natural interactions, the vision, natural language, and also action subsystems of the robotic architecture need to be very tightly integrated to be able to mutually constrain each other. This, in turn, requires concurrent processing of vision, language, and actions where all algorithms must be interruptible and able to incorporate new information incrementally on the fly. It also requires a software framework that allows seamless integration of components and algorithms at a fine temporal granularity. By providing such a tightly integrated system, robots will be able to detect objects faster and more reliably, resolve referential expressions of perceivable referents and ambiguous references more quickly, carry out intended actions more quickly, and achieve much more natural dialogues with humans in everyday environments.

The InSitu project looked at the tight integration of machine vision and natural language understanding. Both of these are tough problems in their own right, especially in the context of autonomous robots, which perform tasks in everyday environments, such as put the yellow cup on the table on the shelf. A human given that task would already, while that sentence is being uttered, follow the gaze or gesture of the speaker and look for something yellow.In InSitu these two sub problems were therefore treated together within a common framework. Cues from (even partially) understood utterances help image understanding by focusing analysis on the relevant parts of the image, e.g., yellow objects in the above example. The visual analysis of a complete complex 3-dimensional scene can be quite time- consuming (many seconds even on current computers). We could show that using situation aware attentional mechanisms together with incremental processing methods the detection of relevant objects in the scene could be significantly sped up. Depending on situation and given task different attention channels are used, e.g. colour, shape or position (... the left cup...).Vice versa visual cues from the scene feed back into language understanding. We could show how to learn object categories in a single shot from utterances such as a medkit is a white box with a red cross on it and verbally describable visual features (cross, red). These learned classes generalize to qualitatively similar objects and allow considerations such asThe cross is green is that still a medkit?. Such semantically deep representations thus go beyond the typical statistical methods, which learn similarity from a large number of examples, without being able to explicitly explain where that similarity lies.Especially important features in this context are those that are functionally relevant, so called affordances, like handles (to pick up) or cavities (the inside of a cup, as a container). In InSitu we developed a taxonomy and systematic approach to describe everyday objects in terms of their affordances, and do detect those affordances in 3D scenes.The methods developed within InSitu were evaluated on 3 different robot platforms at the project partners TU Wien and Tufts University / Boston.