Directional Super-Resolution through Coded Sampling and Guided Upsampling

Sampling and processing images under directionally varying conditions is the key to several modern visual effects production techniques in digital photography and movie making. Light-Field imaging, for instance, seems to have an essential impact for consumer applications, such as digital photography; and has applications in professional domains that rely on robust computer vision and image processing. Light-Field cameras capture spatial and directional scene samples simultaneously to enable digital re-focussing after recording. Since almost twenty years, large camera arrays have been used for visual effects production in cinematography. The simultaneous recording from many different perspectives enables spatio- temporal blending effects during post-production, as seen in the Hollywood blockbuster Matrix (frozen moment camera movement). Today, such visual effects are commonly applied not only in in cinema movies, but also in numerous TV advertisements. Image-based relighting is another post-production technique frequently used in the movie industry. High-speed recordings under sequentially varying lighting directions inside a controlled studio environment, such as a Light Stage, makes the computation of entirely synthetic illuminations of actors possible. Image-based relighting finds applications in many Hollywood movies, such as Spider-Man 2 & 3, Superman Returns, Hancock, The Curious Case of Benjamin Button, and Avatar. All of the above examples have in common that they process images that are recorded under varying (viewing or lighting) directions. They do not require depth-estimation, as this is difficult or even impossible to achieve for realistically complex sceneries. To prevent under-sampling artifacts, however, a substantially large number of images has to be captured and combined. This leads to complex and dense camera- or light-arrays, and to performance constrains that are due to bandwidth limitations. In this project, we want to investigate coded sampling and guided upsampling strategies that, compared to the dense uniform sampling which is state-of-the art for the approaches discussed above, either achieve the same image quality with fewer samples (i.e., less cameras or less light sources leading to lower complexity and bandwidth requirements), or a better image quality with the same number of samples. Thus, our objective is to develop a first guided super-resolution technique for the directional domain that does not rely on scene depth or disparity estimation. It therefore will support realistically complex sceneries, as required for visual effects production in photography and movie making.

Sampling and processing images under directionally varying (viewing or lighting) conditions is the key to several modern techniques that are used in research and in visual effect productions (e.g., image-based rendering or relighting). For example, image-based rendering with light- fields supports various processing operations that are impossible with classical systems. A light-field photograph allows the changing of parameters such as aperture, focus, and perspective after recording; which is impossible with a regular camera image. Image- based relighting supports the computation of novel illuminations on real actors or objects. To support relighting, the model or object is recorded under directionally varying lighting conditions (e.g., with a light dome). To prevent undersampling artifacts, however, a substantial number of samples from many directions must be captured and combined. This leads to costly and complex recording devices, and to performance constraints due to bandwidth restrictions, exposure durations, or budget reasons. In this project, we investigated directional super-resolution techniques for sparse sampling devices (e.g., camera array, light-stage systems and camera drones). We do not rely on depth estimation or precise image correspondences, as this can be difficult or even impossible to compute for realistically complex scenes. We propose non-uniform sample placements and apply upsampling techniques to achieve qualitative results as if significantly more samples have been used (i.e., avoid or reduce undersampling). One of our contributions is the use of local dictionariesextracted directly from the sceneinstead of using global dictionaries that are learned offline from a set of representative pre-recorded scenes. In the course of this project we presented a first guided super-resolution technique and propose an algorithm to design sampling masks for light-field camera arrays and light-stage systems. We extended this idea to apply compressed sensing theory for reconstruction and for finding optimal sampling masks for the application of sparse light-field camera arrays. Moreover, we proposed a new sampling strategy, which neither relies on learned global dictionaries or external database for mask optimization, nor on user-defined guidelines that restrict the number of mask samples as in. We published scientific articles, and presented results wich outperform related work. Our methods show how directional aliasing artifacts in modern visual effects techniques can be improved simply by improving sampling positions. In the future we want to investigate the application of our coding and super-resolution techniques to extremely-wide-aperture light fields, in an FWF follow-up proposal called Wide Synthetic Aperture Sampling. As a first step towards this goal, we implemented a drone- based light-field camera. Furthermore, we showed the possibilities of image-based rendering when such a dataset is available (e.g., retrieving hidden archaeological objects in a forest).