Situated Vision to Perceive Object Shape and Affordances

The objective is to provide models and methods to detect, recognize, and categorize the 3D shape of everyday objects and their affordances in homes. The planned innovations are: (1) We propose the Situated Vision paradigm and develop 3D visual perception capabilities from the view of a robot, its task, and the environment it operates in. (2) We show the generality of the Situated Vision approach by evaluating the performance on different robots at the project partners and in different environments. The Situated Vision approach is inspired by recent work in cognitive science, neuroscience and interdisciplinary work in EU projects: it fuses qualitative and quantitative cues to extract and group 3D shape elements and relate them to affordance categories. Cognitive mechanisms such as situation-based visual attention and task-oriented visual search let the robot execute primitive actions to exploit the perceived affordances. Perception integrates quantitative and qualitative shape information from multiple 2D and 3D measurements. The analysis of the shapes is used to find instances of semantic 3D concepts, such as providing support to objects that can be used to find semantic entities and to learn affordance categories. The system will be tested in three typical home scenarios and clutter in the form of five or more objects around the target object. Four renowned research teams combine their experience to show that the combination of attention (Uni Bonn), categorization (RWTH Aachen), shape perception (TU Wien) and learning (IDIAP) will bring about a big step forward in cognitive robotics.

The scope of the project in the context of research at TU Wien has been the building of algorithms towards the detection of aspects of objects that contribute to the functionality of the object. These aspects are expected to be determined using color and 3D images of the objects. We call these functional aspects affordances or affordance features. For example, when a person observes the concave container part of a cup, this aspect of the cup enables the person to make a determination that the cup can hold or contain solid or liquid substances within the concave cylindrical boundaries of the cup. Such information can also be used to label the object under study as a cup, a mug or a pot, as the case might be. Hence the cup affords contain-ability. In other words, the affordance feature in question is contain-ability. Similarly, it is possible to determine various geometries associated with objects and determine the functionality these geometries give to the objects. At TU Wien, we have developed computer vision algorithms that take color and 3D images as input, determine the parts of the objects in the scene, ascertain the geometries comprising the parts and finally determine the affordances of affordance features that these geometries provide to the object under study. For the first time in the field of affordances, we have been able to develop a concrete and rigorous approach to defining affordances and generating algorithms that help estimate the affordances. The resulting effort is a database that describes over 200 objects commonly found in indoor environments, in terms of their functions (or affordances) using a combination of only 35 affordance feature types. Research in such a function based approach towards recognizing objects is still nascent and our contributions in cooperation with our project partners should enable improve the scope of research in this field significantly. It should be noted that such a cognitive approach to understanding objects and scenes containing the objects based on their functionality is the key towards building robots and artificial intelligence systems that are capable of solving cognitive problems, approaching more realistic and human- like performance. Such technology is envisaged to be deployed in robotic assistants in both domestic and industrial scenarios. Furthermore, the concept of characterizing objects by their function and associating shapes and design of objects by their function is intricately tied to the design philosophy of form follows function that has implications in not just product design but also in architecture and civil engineering.