Bilevel Learning for Computer Vision

Variational methods are among the most successful methods to solve inverse problems in computer vision and image processing. Typical problems are tasks such as image restoration, motion estimation, stereo and 3D reconstruction. Existing variational models in computer vision are mainly hand-designed based on simple principles derived from the intrinsic properties of images. Clearly, these models are often too simple to model the complex physical properties of the visual world. In this project, we make a significant step ahead by considering more complex variational models based on higher-order, non-local and data-adaptive regularization and propose to learn the involved model parameters using optimization methods. The main idea is to learn the parameters of the variational models, such that the solution of the variational model minimizes a certain loss function that measures the error between the ground truth solutions and the solutions predicted by the model. This problem naturally leads to a bilevel optimization problem, where the lower level problem is given by the variational model and the higher level problem is given by the loss function. It turns out that these bilevel optimization problems have many interesting properties, which are still too less investigated in order to make the method accessible for a larger community. Therefore, it is the main goal in this project to develop a unified framework that can be applied to a number of variational problems in computer vision. The unified learning framework will allow us to systematically investigate existing models as well as new models, leading to a better understanding of their advantages and limitations. We expect that the results of this project will lead to new models that can be optimized towards specific applications and image data and hence will perform significantly better than existing models.

The main objective of the BIVISION project was to combine classical variational approaches in image processing with modern methods of machine learning in order to learn richer and more accurate models for image processing from a large number of natural images. The combination of machine learning and variational models naturally leads to so-called bilevel optimization problems, which represent the nested combination of two optimization problems. The parent optimization problem is given by the learning problem while the child optimization problem is being defined by the variational model. In the course of the project, numerous novel numerical algorithms and methods were developed to learn better models from the image data. One of the most important developments of the project were the so-called variational networks, which represent a hybrid between classical variational models and modern deep convolutional neural networks. Variational networks are comparable in performance to modern deep convolutional neural networks, but allow a better understanding of how their individual layers work. As part of the project, the developed models were used in a wide variety of applications in computer vision, image processing and inverse problems. As an example, we highlight the calculation of high-quality magnetic resonance images from noisy and subsampled measurement data. This work was carried out in cooperation with New York University and published in a renowned journal. Future work will be devoted to the theoretical analysis of variation networks. The goal will be to develop a theoretical backbone that allows to give regularity statements and reconstruction guarantees that correspond to those of the classical calculus of variations.