Semi-Autonomous Aerial Vehicles for Augmented Reality

In recent years, mobile touch devices are increasingly replacing or supplementing desktop computing. However, touch devices require direct physical and visual attention, both of which are the scarcest resource in mobile situations. Research on Augmented Reality (AR) and Wearable Computing has long attempted to address these issues with alternative user interfaces that augment the users perception of the real world. Traditionally, this has been achieved by instrumenting the user with head-worn displays, tracking devices etc. As a result, trade-offs concerning ergonomics, interactive possibilities and data richness were unavoidable. We propose a radically new type of natural user interface (NUI) that brings together elements from robotics, AR, and ubiquitous computing. We intend to build the users flying organizer (UFO), a semi-autonomous micro aerial vehicle (MAV) equipped with pico-projectors and cameras that allow for projected interactive pixels anywhere in the users environment. Whilst MAVs have been explored in the context of autonomous flight, this project will for the first time explore how these can be made fully interactive, working in synergy directly with users in real-world environments to enable new application scenarios, including: 1) The spontaneous visualization of data directly on physical surfaces around the user, turning the environment into an ad-hoc smart space augmented with graphics and interactive possibilities. The planned system allows UFOs to navigate freely in the environment, projecting content where needed, without user instrumentation. 2) New forms of mobile computing. Allowing a user to see around corners or into otherwise physically unreachable locations via the drone, for example in search and rescue scenarios, whilst maintaining access to digital information. 3) New forms of teleconferencing, where a remote participant may leverage the drone to experience the users surroundings and to project information into the environment to jointly perform a task that otherwise would require co-presence. The proposed project will have impact on fundamental human computer interaction (HCI) research. The primary outcomes will include the development of new insights, methods and tools for NUI interaction, their realization in novel algorithms (e. g., new gesture recognition techniques), their implementation as open source software and hardware (e. g., MAV platform and sensing technologies), and their application in challenging real-world scenarios. The success of the project is backed by a significant amount of preliminary research results by the PIs and their research groups. The application of the techniques pursued in this proposal can potentially have immediate impact on emerging areas in the marketplace and hence society.

In recent years, mobile touch devices are increasingly replacing or supplementing desktop computing. However, touch devices require direct physical and visual attention, both of which are the scarcest resource in mobile situations. Research on Augmented Reality (AR) and Wearable Computing has long attempted to address these issues with alternative user interfaces that augment the users perception of the real world. Traditionally, this has been achieved by instrumenting the user with head-worn displays, tracking devices etc. Subsequently, trade-offs concerning ergonomics, interaction and data richness were unavoidable. As a result of the UFO project, we were able to propose a new type of natural user interface bringing together elements from robotics, AR, and ubiquitous computing. In a first step, we introduced the so called Micro-Aerial Projector, a semi-autonomous micro aerial vehicle (MAV) equipped with pico-projectors and cameras, which can, either physically or virtually, project interactive pixels anywhere in the environment without encumbering the user. Whilst MAV technology has been explored in the context of autonomous flight, this project, for the first time, investigates interaction with highly mobile projection devices. New application scenarios include: 1) The spontaneous visualization of data directly on physical surfaces around the user, turning the environment into an ad-hoc smart space, augmented with context dependent projection screens and interactive possibilities. The system allows to navigate freely in the environment, projecting content where needed and adapting to the environment (keeping distance to the user) without additional instrumentation of a remote user. 2) New forms of mobile computing, augmenting the users field of view within the help of the projection capabilities of MAP. User can see through walls or into otherwise physically unreachable locations, and naturally explore them. For example, in search, rescue or inspection scenarios, access to digital information is maintained. Additionally the project established a complete new research- and design space, the so called droneSpace, at Graz University of Technology. The core component is an autonomous MAV flight system, including a flight arena tracked by a highly accurate motion tracker, a self-designed MAV-prototype and the according software framework. Thus, we were also able to put up a sustainable basis for interdisciplinary research between the fields of Augmented Reality, Robotics and Computer Vision.