Panorama Light-field Imaging

Since the invention of the first camera, images always have been two-dimensional matrixes of pixels. Even though 3D scene points emit varying light rays in different directions, the lens and the sensor of cameras integrate them to a single pixel whereby loosing most of the scene information. What if instead of capturing, storing, processing and displaying only a single color per pixel, each pixel would consist of individual colors for each emitting direction? Images would no longer be two-dimensional matrices but four-dimensional ones (storing spatial information in two dimensions, and directional information in the other two dimensions). This four-dimensional structure is called light field. Light fields have the potential to radically change everything that we relate to images - from photography, over displays to image processing and analysis. Light-field photography extends conventional digital photography by multi-perspective information that enables 3D processing and viewing (including synthetic refocusing, 3D depth reconstruction, large depth-of-field photography, auto-stereoscopic viewing, etc.). We believe that multi-perspective imaging and display will play a major role for the near to mid-term future of digital cameras and displays. After more than fifteen years of research and development in light-field imaging hardware, and with increasing sensor resolution of digital cameras, light-field photography has now greater potential than ever to succeed digital photography. Several vendors already offer compact commercial light-field camera systems. Light-field processing techniques for photography purpose, however, are still limited mainly to synthetic refocusing and perspective changes within the synthetic aperture range. The application of digital processing techniques to captured images is one reason why digital photography became successful. Among many other features, software algorithms allow adjusting brightness, contrast and color, enable rescaling and cropping, or make the recording of large panoramas possible that exceed the camera`s optical field of view. Applying such image processing techniques to light fields, however, is in many cases not straight forward. The reason for this is that the outcome (i.e., a light field) must not only be spatially consistent, but also directionally consistent. Panorama imaging techniques already became an integral part of digital photography - often being supported by camera hardware today. In this project, we aim at investigating and developing novel light-field processing algorithms that make the recording and the viewing of panorama light-fields (i.e., large field-of-view panorama light-field recordings) possible. As for other image processing techniques, a straight-forward application of panorama imaging to light fields is not possible, since spatial and directional consistency has to be ensured. The impact of panorama light-field imaging to light-field photography is comparable to the impact of panorama imaging to digital photography. It holds potential to become an integral part of upcoming light-field cameras and of light-field photography in general.

Panorama imaging techniques allow recording photographs that exceed the camera's optical field of view. They already became an integral part of digital photography -- often being supported by camera hardware today. In this project, we focused on enabling panorama capturing with light-field cameras. Light-field photography extends conventional digital photography by multi-perspective information that enables 3D processing and viewing (including synthetic refocusing, 3D depth reconstruction, large depth-of-field photography, auto-stereoscopic viewing, etc.). Several vendors already offer compact commercial light-field camera systems, and with increasing sensor resolution of digital cameras, light-field photography has now greater potential than ever to succeed digital photography. The application of digital processing techniques to captured images is one reason why digital photography became successful. Applying image processing techniques like panorama stitching to light fields, however, is in many cases not straight forward. The reason for this is, that the outcome (i.e., a light field) must not only be spatially consistent, but also directionally consistent. In the course of this project we introduced a first panorama imaging method that allows combining multiple light fields captured by a regular commercial light-field camera. Furthermore we investigated methods for rendering the resulting large light fields in real-time on consumer graphics cards. Light fields can not only be captured by specialized hardware, but also by recording multiple images with a moving regular camera. Todays smartphones are the optimal tool for this. We developed an algorithm which guides novice user while capturing different kinds of light fields with their smartphones in a matter of minutes. Finally we investigated a general light-field deformation method. It improves the computation of panorama light fields in case of imprecise camera movement, but also lays ground for other light-field processing algorithms like retargeting or artistic deformations. The impact of panorama light-field imaging to light-field photography is comparable to the impact of panorama imaging to digital photography. It holds potential to become an integral part of upcoming light-field cameras and of light-field photography in general.