3D Video Analysis for Interactive Multimedia Applications

The area of video products and applications is expected to experience a significant growth over the next couple of years. New and improved methods for the efficient storage and retrieval, transmission, and analysis of video material need to be developed for a wide range of applications such as the entertainment industry, educational and cultural initiatives, mobile communications, as well as medical and safety applications. In our project, we concentrate on video processing tasks that arise in the context of interactive multimedia applications. In particular, we investigate the need for user interaction in fundamental video editing steps such as the insertion or deletion of objects, or the combination of videos from different sources ("video compositing".) These tasks currently require a significant amount of user assistance. A prerequisite for their automation is the recovery of a 3D model from the video scene, which afterwards provides the framework for the proper handling of occlusions and perspective distortions. During the first part of our study, we perform experiments with two and more video cameras in various spatial arrangements in order to better understand the advantages and limitations of different stereo configurations on the subsequent 3D analysis. The main part of our investigations then deals with the development of suitable video processing algorithms for the extraction of spatial information from a dynamic video scene. In this context, we focus on methods for the automated segmentation and tracking of video objects and their combination with stereo- based depth analysis. The final goal of the project is to generate a high-level description of the video scene that relies on the identification of semantically meaningful video objects and their location in 3D space. Such a representation provides the basis for the subsequent encoding of the video content according to the recent MPEG-4 and MPEG-7 standards for video compression and description.

The project `3D Video Analysis for Interactive Multimedia Applications` deals with the reconstruction of 3D scenes and motion analysis from videos, with the goal to automatically extract people or objects from the background. Video object segmentation is an important prerequisite for a variety of multimedia applications such as video editing (insertion, deletion, or modification of objects) or the combination of real and synthetic scenes (video compositing, augmented reality), where the 3D reconstruction is required for the proper handling of occlusions, shadows, and perspective distortions. Our research results can be employed for video coding (e.g., MPEG) and are of particular interest for potential commercial exploitation in the context of upcoming video/TV user scenarios such as `Interactive 3D TV`. As part of this project, we have developed a new stereo algorithm that delivers excellent reconstruction results (i.e., depth maps), also along object boundaries and in occluded regions, which are usually difficult to reconstruct. The high quality of our stereo maps was confirmed by quantitative evaluation in a benchmark test, where our algorithm outperformed most other state-of-the-art stereo matching techniques. Further research dealt with the development of novel algorithms for motion analysis (optical flow) and subsequent motion segmentation based on color processing and clustering image regions that undergo similar motion. We illustrate the subsequent utilization of the motion-segmented video objects for video compositing tasks using standard MPEG test scenes. We employ our reconstructed 3D scenes and motion-segmented video objects in a variety of novel applications in the field of image-based artistic rendering. Our stereo-derived depth maps allow us to automatically produce computer-generated sketches with different drawing styles from natural images as input. Also, we show how a combination of newly developed computer vision and graphics algorithms can be utilized for stylized motion depiction from videos, to achieve an aesthetic impression similar to comics produced by hand. As a further application, we suggest the computer-based generation of stereoscopic paintings from real stereo images or videos, for example, in imitation of an impressionist painting style (image-based stereoscopic painterly rendering).