Space-Time Representation and Recognition in Computer Vision

This project shall investigate spatial and temporal relations in video in a principled manner. We will develop a novel representation, called ``space-time volume of interest`` VOI, that will be used consistently to represent various entities in video data, for instance independent foreground motion, background motion patterns, and repetitive patterns in space-time. These entities will be described by extended space-time oriented energy filters, and the descriptions will be utilized to categorize video textures, dynamic scenes, types of camera motion, independently moving foreground objects, and activities. The basic, individual representational units, i.e., individual VOIs, shall be integrated to complex, compound space- time models that cast votes for particular spatial and temporal locations as well as scales. The project is coarsely structured into four work packages, addressing the VOI representation itself, extraction of local space-time descriptors, learning and long-term space-time description, and online detection/categorization in videos. With this research project, we expect to contribute to a better representation and understanding of the ``deep structure`` of visual space-time. The novel concepts developed in this project will initiate a variety of further basic research in this field, as well as encourage new applications in video analysis, surveillance, and autonomous systems.

What happens where and when in a video? How to track, represent, and classify objects, and actions? What are major applications of such novel methods? The project has addressed these key questions from August 2014 July 2017. In these three years, the field of Computer Vision has seen a paradigm shift from previously used explicit modeling and specific algorithms to now popular implicit representation in deep, convolutional neural networks (deep ConvNets). These new methods largely outperform all previous algorithms, with a general increase in system performance and reliability that puts significant applications within reach. This paradigm shift includes a shift in research focus towards architectures of deep networks, machine learning, and image- and video-databases for training. We have achieved several remarkable results, in close collaboration with York University, Toronto (explicit representation of space-time using space-time oriented energies) and with University of Oxford (implicit representation in ConvNets, in particular novel two-stream architectures that can model and combine appearance and motion in video). Our results include the release of a new benchmark video dataset for dynamic scene recognition, and a number of novel ConvNet architectures, which we trained to establish a new state-of-the-art in dynamic scene recognition, human action recognition, and object detection and tracking in video. The focus of this basic research project has mainly been on the development of novel methods. We succeeded in publishing an outstanding number of papers in top ranked Computer Vision conferences and journals. Beyond these purely scientific achievements, there is a number of highly relevant applications of our methods in areas such as autonomous driving, automated video analysis and annotation, video surveillance, and many more. Towards the end of the project, we also achieved a major breakthrough towards better understanding and analyzing of what is happening inside deep neuronal networks. For the first time, we now can visualize the learned representations of the motion stream in ConvNets, which shows us what has been learned and how it is represented in which units at which layers in the net. This also leads to an improved analysis of problems and failure cases.