Autonomous Exploration of Unknown Environment by Fusion of Stereo Vision and Inertial Sensors

Tracking is required in many important applications, e.g. navigation, 3D human-computer-interaction, and surveillance. Most often, information has to be obtained in "real-time". Many kinds of sensors have been used for this purpose in the past: sonars, radar, laser range finders, GPS, electronic compass, so-called "magnetic trackers" in virtual reality, computer vision, transponders, just to mention the most common techniques. This project investigates a new, generic approach to real-time inside-out tracking (a system which is mounted to a mobile object with the main purpose to track the position and orientation of the object itself) using a combination of "smart sensors". We plan to use a sensor suite consisting of a fixed, calibrated stereo rig (with directly addressable multi-window CMOS cameras) together with an "inertio-tracker" based on accelerometers and gyroscopes. These two sensor types provide complimentary characteristics: visual sensing is very accurate at low velocities while inertial sensors can track fast motions but suffer from a drift particularly at low velocities. This inside-out tracking system will be subjected to arbitrary motion (on a mobile robot, carried by a person, ...) and shall operate in cluttered multi-object scenes with multiple and independent motion and some static, but 3D background. The primary goal is a reliable reconstruction of the trajectory of the system itself, as well as the recovery of 3D structure required for a successful tracking (salient landmarks). A further goal is to trace the trajectories of all detected other moving objects. Similar to the perceptive capabilities of a human, the system shall operate autonomously, without requiring additional information about its localisation and pose (like GPS or artificial landmarks). New CMOS camera technology will be used to gain an order of magnitude in optical tracking speed, and fusion of a suite of new accelerometers and gyroscopes with vision will provide robustness in cases of fast motions as well as the superior accuracy of vision-based tracking. The new tracking techniques will be applied to VR/AR applications at TU Graz and to mobile robot navigation at TU Wien. The goal of the proposed project is beyond previous work on "tracking" of 2D motion in images or 3D structure from motion captured by stationary observers. We aim at the continuous tracing of a moving object`s position and orientation in space and time, and we propose a way of achieving this goal based on sensory information captured by the moving platform itself.

The aim of this project was to research into new sensors and algorithms for online structure and motion analysis. More specifically, camera(s) are mounted on a sensor head and are used to measure visual landmarks in a scene. These vision-based measurements are complemented by inertial sensors. Finally, visual and inertial information is fused to obtain the current pose and the trajectory of the sensor head over time. To be applicable for online analysis, all algorithms should work in real-time, sensors should be sufficiently fast, and calculations should be incremental (as compared to off-line batch-processing, e.g. in photogrammetric bundle adjustment). The main results of the project can be split into two parts: The development of a new high-speed camera, and the development of new, incremental structure and motion algorithms. Once a landmark has been detected in an image, any motion of the camera will lead to a 2D displacement of the landmark in the image plane. The basic idea behind our new high-speed CMOS camera is to read only small windows from the image sensor, instead of whole frames. We have developed the necessary hard- and firmware to address arbitrary sub-windows of a CMOS imaging sensor in a fast and efficient manner. The new camera system works either in a monocular or in a stereo setup and has a standard USB-2 interface. We achieve "frame" rates of up to 2.500 windows per second (as compared to only 5 full frames per second), which allows us to track visual landmarks very fast and reliably. Our new structure and motion algorithm fuses the data from the CMOS sensor with the inertial data from the IMU (Inertial Measurement Unit), which typically operate at different and possibly varying rates. The system can then move into a new environment from a single known starting pose and simultaneously estimate the egomotion of the sensor rig and the unknown structure of the scene. Data fusion achieves the estimation of the egomotion of the sensor rig and takes into account cases such as the dynamics in the scene (occlusions, foreground motion), loss of correspondence of visual features, convergence (in case of non-linear models), fast camera motion (motion blur and especially rotations), and additionally estimates the 3D position of newly detected landmarks in the environment. Such sensors and algorithms have a high potential and might be applied in a number of relevant future systems, including mobile computing, location and pose aware communications, vehicle driver`s assistance, and autonomous navigation.