V-MAV: Cooperative micro aerial vehicles using onboard visual sensors

The overall aim of the project is to advance the capabilities of visual controlled MAVs in the areas of flight behavior and autonomy, cooperative operation, cognitive abilities and in addition to decrease the size of such an MAV. Advances in these areas would enable new fields of applications for MAVs and path the way to further research topics in mobile robotics. The proposed research proposal is structured into three work packages: 1. Visual-inertial MAV pose estimation and localization using multi-camera systems 2. Embedded vision algorithms for dynamic flight of small scale MAVs 3. Methods for cooperative visual localization and semantic mapping Work package 1 will investigate the suitability of multi-camera systems for 6DOF pose estimation and localization for MAVs performing dynamic maneuvers. This will include the development of visual-inertial pose estimation algorithms exploiting the advantages of multi-camera system geometries. Work package 2 will investigate embedded computer vision algorithms to facilitate dynamic control and flight as well as a further miniaturization of MAVs. For this, specific components of the visual control system will be moved to dedicated embedded processors to achieve the necessary high-frame rates for dynamic flight. Work package 3 will investigate cooperative operation of MAVs focusing on cooperative visual localization, mapping and cognitive scene understanding and interpretation. In cooperative operation MAVs should be able to share their individual knowledge of the environment and incorporate knowledge of others with the effect of improving environment mapping and the self localization process. An important part of this research package is cognitive scene understanding. The MAVs should make use of object detection and classification methods to generate a semantic description of the environment to produce a semantically annotated 3D environment map and also to use this meta-information to improve the mapping process (e.g. adapt parameters based on the semantics) or the localization process. The proposed project will combine the competences of the three involved partners, ETHZ, TUM and TUG. All the three partners have year-long experience in vision controlled MAV through various projects and performed ground breaking work in this area. The common project will ensure the utilization of the combined expertise of the partners.

In the DACH project V-MAV, the aim was to improve image-based algorithms used to control Micro Aerial Vehicles (MAVs). The three partners, TU Graz, TU Munich and ETH Zurich, worked on localization and pose estimation for MAVs using multi-camera systems and visual-inertial systems (i.e., camera systems also using accelerometer, gyroscope and compass data), embedded image processing algorithms (i.e., usage of specifically designed hardware for image processing, with which it is possible to make the MAV smaller and the image processing faster) and on mapping of the environment using images taken with MAVs. The mapping part also made use of additional scene meta-information (e.g., semantic information which part of the scene is a tree, which part is a house) in order to improve the mapping result. In our project part, we mainly focused on visual localization and mapping. Our investigations in visual localization resulted in an image-based localization method, which runs in real-time and, hence, can be used for navigating an MAV. Our system mainly uses vertical lines to compute the camera movement. Such lines occur very often in man-made environments (e.g., at windows, doors, building outlines) and, hence, can be used especially for such environments to improve the localization results. In order to detect vertical lines in a fast way, we used an Inertial Measurement Unit (IMU), which delivers accelerometer and gyroscope data, to detect the gravity direction and consecutively detected lines which are parallel to the gravity direction. In a next step, we incorporated the IMU information directly into our localization algorithm in order to improve the localization quality. In the area of 3D mapping, we investigated in several reconstruction techniques to create compact models, which can be transmitted easily via network, and visually appealing 3D models especially for urban environments. Our first algorithm delivered very compact and visually appealing representations of buildings and specific scene structures. To get similar results for arbitrary urban environments, we developed an additional approach, which detects planes in the scene and makes these plane surfaces selectable in the final reconstruction process. By adjusting the reconstruction parameters, it is possible to adjust how precisely the reconstruction should follow the planes. Additionally, we used semantic information (i.e., which parts in the image are trees, buildings, streets) to improve the 3D reconstruction result. We used artificial intelligence methods to segment images into different semantic classes. Then, using this semantic information, we adjusted the 3D reconstruction parameters depending on the semantic class (i.e., a façade should be planar, a tree should have a smooth surface) and showed that this improves the reconstruction result.