Augmented Reality by Example

Augmented Reality (AR) by Example is a novel approach to generate AR applications from existing video and image demonstrations. Previous techniques create AR applications using a time-consuming process, which requires skills with 3D modeling and animation software, in addition to knowledge of the technical components of an AR system. While any AR application will benefit from a simple authoring process, this project will mainly target the generation of AR applications for knowledge presentation in tutorials. While traditionally images have been used to provide visual instructions, with the success of video sharing platforms a large body of video tutorials is additionally available. Both, image and video tutorials effectively convey complex motions, but are difficult to follow precisely because of their 2D nature. AR tutorials have been revealed to be more effective. This research projects brings the advantages of 2D and 3D instructions together by automatically creating three-dimensional AR tutorials from conventional 2D data. Unlike previous work, we will not simply overlay video, but synthesize 3D registered motion from the input data. Since the information in the resulting AR tutorial is registered to 3D objects, the user can freely change the viewpoint without degrading the AR experience. This is achieved by investigating a number of different techniques. First, we have to extract the author`s 3D environment and all 3D motion from the input data. Second, we need to provide tools for editing and retargeting the resulting 3D scene to the user`s current environment. Third, we will develop comprehensible visualization techniques, specific for instructions in AR environments to effectively communicate the instructions extracted before. The research is complemented by qualitative and quantitative evaluations of the effects the investigated techniques have on users. This research project has great potential in applications concerned with crowed sourced training and teaching.

Augmented Reality (AR) by Example was a research project investigating novel approaches for AR applications based on video and image data. The project was mainly targeting knowledge presentation in AR tutorials. Traditional image and video tutorials effectively convey complex motions but are difficult to follow precisely because of their 2D nature and their missing link to the user's real world environment. Since AR tutorials have been revealed to be more effective, the goal of the project was to provide novel tools to make 2D image and video data more accessible in 3D AR environments. This was achieved by investigating approaches for AR scene acquisition, sequence detection, motion visualization, and retargeting of captured 2D data to 3D AR environments. In particular, a novel approach for authoring AR assembly tutorials from video demonstrations has been developed. The approach makes use of a 3D model and the corresponding assembly graph of the assembly to generate a 3D animation, which is annotated with corresponding video sequences that have been extracted from the input video. This approach was subsequently refined by also deriving the 3D model and the assembly graph from an RGB-D input stream. Furthermore, to coherently render the modeled and the real environment in AR, this project has developed a series of neural networks, which are able to mimic the characteristics of the real camera used in video-see through AR applications. To avoid the need for a 3D model, we have also developed a novel approach for authoring and viewing AR instructions based on an image database. Thus, approaches for guided capturing, editing and display of light fields on mobile devices were researched. This has been demonstrated to well support applications to remote AR assistance in complex and unprepared environments. Recent advances in free viewpoint rendering have furthermore enabled to extend the developed approach from images to multi-view video representations. In the context of this project, a light field video capturing and rendering approach has been implemented. To research the impact of 3D AR video tutorials in comparison to traditional model based AR tutorials, we have furthermore developed an approach to automatically generate interactive AR tutorials for users at different skill levels. We focused on generating guitar tutorials as it provides a challenging application case to learning motor skills. To reduce perceptual issues while following AR tutorials, we have also developed a novel video-based scene representation that supports focus cues. Our approach is based on a focal stack representation, as they naturally support defocus blur. Thus, we introduce a pipeline for capturing, rendering and blending of focal stack images in real-time frame rates. We demonstrate that this representation is able to efficiently drive layered displays to present focus cues to an AR user.