Making Sense of Objects from Exploratory Robotic Interaction

Whether it is to prepare food items for cooking, to serve meals or to tidy up a table many robotic assistance tasks may be viewed as rearranging the environment, for which they need to move objects between locations. Robots must be able to carry out such tasks in dynamic environments, such as homes or offices, where known objects move and change over time, while new objects are introduced regularly. For example, in a kitchen, a bag of flour may deform as it is emptied or a new brand may come in a completely different packaging. However, such changes in the objects appearance are difficult to handle for current robotic perception approaches. At the same time, changes in the objects shape also complicate robotic interaction. Adapting to and learning how to cope with these changes poses a significant challenge to robots everyday operation. Our ambition is therefore to set the ground for continuous and autonomous robotic learning of objects and interactions therewith, allowing robots to make sense of their environment through exploration. On one hand, exploratory interaction generates situations that allow the robot to progressively learn causal relationships to object properties that it would not observe otherwise. On the other hand, when it encounters a situation that is beyond the limits of its current capabilities, the robot may revert to exploratory interaction, allowing it to robustly deal with change and to adapt to novel tasks more quickly. For example, based on the exploratory interaction with different containers, a robot may learn to adapt to their different shape, weight or stiffness. In contrast to simply learning to interact with each individual container, the robot may learn a relationship between the containers shape, physical properties and the way it needs to interact with them to fulfill its task. By learning such causal modules from exploration, the robot attains an understanding of concepts that are disentangled from a specific object or environment. This will be demonstrated in variations of the rearrangement task with different goals. Exploiting increasingly complex information about the objects, the robot will carry out dynamic interactions that would not be possible without the proposed robotic sense-making. As such, the investigated exploratory approach will decrease the human effort in teaching robots to interact with novel objects, while increasing the robustness of robots perception and interaction to variation of objects` appearance and physical properties.