Energy Functions for Global Stereo Matching

During the last few years, stereo matching has experienced a significant advance with the introduction of new optimization algorithms. Energy minimization methods based on these optimization schemes currently show the best performance in stereo computation. However, while a lot of research effort has been put into the optimization problem of the energy minimization approach, the fact that the energy functions under consideration might represent an unsatisfactory model for the stereo problem has often been ignored. In the proposed project, we aim at pushing the state-of-the-art in stereo vision by investigating and improving the modelling component of energy minimization techniques. One major contribution to the stereo community is that we will run a competitive performance evaluation among energy functions that have been proposed in the literature. Energy functions are typically combinations of several terms that are motivated by the same idea, but differently implemented in each approach. Moreover, the resulting energy functions are minimized using different optimization algorithms. It is therefore difficult to judge why one approach outperforms the other. In the proposed project, we will implement a framework that unifies several energy functions and accounts for their minimization. This framework will serve as the basis for a benchmark test, in which we will use image pairs of real scenes along with ground truth disparities. The goal of our experiments is to identify which components of an energy function result in a performance improvement and which do not. This will lead to a new and deeper understanding of current energy functions and point out those factors that show the highest potential for further improvement. In the second phase of the project, we will use the knowledge gained in our evaluation study to develop novel energy functions. These energy functions will be designed to deliver high-quality disparity maps that improve over the current state-of-the-art. These high-quality disparity maps are vital for a variety of applications, ranging from quality assurance, robotics and virtual reality to promising applications in the entertainment industry such as novel view synthesis.

In this project, we have carried out a systematic analysis of stereo matching algorithms regarding their accuracy and potential for improvement. The gained insights have led to the development of novel matching algorithms that have shown excellent performance in an international benchmark test. The project results have obtained several awards and have been presented to the public in scientific media reports (for example, by Austrian TVs Newton program). The key task of a stereo matching algorithm is the automatic identification of corresponding points in the left and right image of a stereo pair. The point correspondences are then used to generate a (3D) depth map of the observed scene. Global matching techniques first set up a 3D model of the scene by formulating a suitable energy function. The goal of the subsequent optimization step is to find that solution (i.e., depth map) which best explains the scene visible in the left and right stereo view, while being consistent with the previously defined scene model. In our work, we put an emphasis on the modeling step, since it was found to offer more potential for improvement than the optimization procedure. As part of the project, we have analyzed the scene modeling step regarding various aspects. In particular, we have investigated the influence of color and the selected color space on the quality of the 3D reconstruction. A special challenge for stereo matching algorithms is the task to capture suitable features in the surrounding of a regarded image point, while at the same time the search area should not be extended unnecessarily, in order to save computation time. In this context, we have developed novel matching techniques based on efficient tree structures and appropriate weighting techniques introduced into the search process. Furthermore, we have suggested a method that enables the highly precise reconstruction of object contours. Our technique is able to cope with subtle color mixture effects that tend to occur along object boundaries and in the surrounding of fine image structures (e.g., hairs), where conventional stereo algorithms typically fail. The obtained high-quality depth maps are vital for a variety of applications, ranging from quality assurance, robotics and virtual reality to promising applications in the entertainment industry, such as novel view synthesis for 3D TV scenarios.