Photo-Guide: Image-Based City Exploration

The core idea of the project is the enhancement of current state-of-the art navigation systems by visual information obtained from geo-referenced photographs. The aim is to establish a suite of tools together with algorithmic foundations that will be essential for any large scale image-based navigation project. The main hypothesis of this project is that in the case of navigation, especially for pedestrians, the first-person perspective is intuitive and leads to natural orientation. In the project we will research how to generate first-person views on top of databases of simple ground-based photographs and classical overview maps. The ultimate idea is a system which allows the user to navigate through a city and even through interiors of buildings guided by the aid of annotated first-person views. Furthermore, the actual creation of the routes shall happen automatically on demand. So while there have been some inroads into the problem of using images to enhance navigation systems, the idea of using image collections to create pedestrian views is largely unexplored. We want to research three main scientific challenges: modeling of a geo-referenced image database, modeling of a semantic navigation space, and finally route visualization using view mapping and semantic labeling. The first mentioned direction can build on several papers of recent work. However, none of these publications provides a satisfying solution to the actual point, which is the combination of exact geo-referenced maps with oriented and registered images. We will address explicitly this problem and expect to come up with novel contributions to its solution. Perhaps the biggest challenge in the project is the modeling of the semantic graph on the top of the map. This space will combine (geo) spatial objects with semantic meaning. It will raise the map into a 3d space and open a new field for further applications. For the third research focus we see challenges in advanced view mapping methods, which have recently gained interest due to the massive progress in image-based methods. Such projections combined with innovative algorithms like content-aware resizing, Poisson image editing, graph-cut segmentation and paint-selection open the door for many new applications. In this context we expect to contribute to the state-of-the-art with novel algorithms for view- mapping in combination with image annotations. Examples of fields that can benefit from the proposed approach are tourism and general entertainment industries, city planners, local governments, simulation and security training, emergency management, civil protection and disaster control, as well as driving simulation to name but a few. Ultimately, the following scientific fields can profit from the proposed research: computer graphics with image processing, computer vision, computational photography, pattern recognition, photogrammetry and remote sensing, cartography, computer aided design, geo- sciences and mobile-technology. We think that our results can lead to a successful commercialization within three additional years.

Geographical navigation systems have become ubiquitous in recent years, mostly through the wide availability of interactive routing applications on mobile devices. Currently, these systems provide either very basic top-down map views, or very coarse 3d reconstructions from aerial photography. Some systems like Google Streetview also show ground-level imagery in selected locations. There is, however, a lot of room for improvement in the visual quality of these navigation systems. Therefore, the Photo-Guide project investigates different methods to use real-world photographs to aid navigation applications. The project comprises two main areas: first, the reconstruction of 3D models from photographs, which can be used especially in ground-level 3D navigation applications; and second, the improvement of classical map-based navigation scenarios, both through better arrangement of information, as well as through photographs.Regarding the first area, we developed novel interactive methods to efficiently create detailed 3d models of real-world objects. In particular, we show that the right combination of user input and automatic computation allows a better and more efficient reconstruction of building facades than purely automatic or manual models. We also investigated the completion of missing data, where we leveraged tensor completion for incomplete image data and structure completion for missing geometric data. Furthermore, we developed novel methods for the creation and modification of texture maps that add detail to the reconstructed models. To this end, we implemented a framework for interactive image color editing, exploited graph cuts to create seamlessly textured models, and also experimented with Gabor noise to create a more sophisticated appearance.The resulting models can be used to show directions in first-person-views and are therefore predestined to be used in navigation systems. Furthermore, they can serve as proxy geometry for augmented navigational hints, thus avoiding occlusion problems between geometry and such hints. They are also interesting for architects, archaeologists and cartographers.Regarding the second area, we developed a novel system that combines classical maps and photos for navigational purposes, both for the automatic generation of tourist brochures as well as for dynamic path exploration on mobile devices.In summary, this project broke new ground in using photos in a navigational setup as well as the accurate reconstruction of real-world objects, thereby considering the geometric as well as the color information, and provides fertile ground for further research, as well as new state-of-the-art methods for applications.