Superhumans - Walking Through Walls

In our research project we will develop new data structures and real-time methods that allow users to intuitively feel and manipulate 3D scan data already while acquiring it. Combining virtual and augmented reality displays with portable range sensors can permit to immerse users into an experience of 3D data that was just captured live. One challenge is to design operations which clean, transform and structure the raw data fast enough to provide a lag-free user experience. The other is to structure the data in order to enable new ways of interaction with the scene, de-coupled from physics-based metaphors like walking or flying. Through our project, we introduce a paradigm change for navigation in virtual and mixed environments. Furthermore, we expect the proposed data structure and implemented methods to enhance the speed of human-computer interaction in such environments. The expected advances in conducting virtual experiences directly contribute to several other basic and applied research efforts. Applications include, but are not limited to, medical healthcare via 3D visualization of 2D CT scans, geology and geophysics via structure measurement and analysis of LIDAR data of surfaces, engineering and prototype design (e.g., cars and aircrafts), as well as physics, biology and astronomy. Other possible applications loosely related to research include military training, crime-scene construction, and tourism. We propose a new view-dependent data structure that permits efficient connectivity creation and traversal of unstructured data, while classifying occlusions at no extra cost. Based on this data structure, we will develop new methods for fast surface recovery, collision detection, as well as browsing and interactive manipulation of dynamic environments. The new data structure will also allow quick access to occluded layers in the current view. This enables new methods to explore, manipulate and edit 3D scenes, overcoming interaction methods that rely on physics-based metaphors like walking or flying. In a way, this lets us lift interaction with 3D environments to a superhuman level. The special contribution of our project is that we cut short the time required to transform scanned 3D data into a structured form which permits browsing through the scene, as well as touching and editing the reconstructed surfaces. Post-doc Dr. Stefan Ohrhallinger will be the principal investigator together with Prof. Dr. Michael Wimmer, head of the Rendering and Modeling group of the Institute of Computer Graphics and Algorithms, and PhD student Mohamed Radwan will also work on this project.

The main outcome of our project are ground-breaking results in real-time visualizing and exploration of large 3D scans with superhuman capabilities as well as results in curve and surface reconstruction that will permit many novel applications. Fundamental results: We exploit the inherent occlusion structure from displaying point clouds in order to quickly act and move through these layers "super-human"-style in large scenes. We sample curves by ordered points with a density corresponding to feature size so that all details are represented fairly. Our sampling method enables evaluating the performance of curve reconstruction algorithms, especially for those without theoretical guarantees since it tests these limits using a brute-force approach. Ground-breaking performance: We designed a deep learning method for surface reconstruction of general shapes that is trained on both the global shape and small local patches and yields impressive improvements even for shapes that were not in the training set. This publication has been cited 140 times in three years. Relevant algorithm speed-up: We implemented an algorithm that can visualize point clouds with billions of points in real-time, about a magnitude faster than prior work. Also, we can reconstruct closed surfaces from such point clouds, also an order of magnitude faster, and with much less memory than the state-of-the-art. Real-world application: Our curve-sampling method permits accelerating simulations such as animations on, e.g., hand-drawn curves, or heat simulations, since only the small details need to be sampled densely. The depth data structure that we developed enabled the design novel real-time user operations such as revealing partially occluded objects, or scrolling through layers of occluding objects, e.g., walls in a building. As an example application we showcased a 3D scene exploration framework that enables fast, more sophisticated interactions with point clouds rendered in real time and paves the path for usage in XR applications. Consolidation of the curve reconstruction field: Together with the other eminent researchers in the domain, we created a comprehensive survey of all state-of-the-art algorithms that includes a benchmark using a combination of existing and new data sets. We held two tutorials at major conferences to disseminate this knowledge. Outlook on continuing work enabled by our results: We applied a little-used connectivity graph to curve reconstruction, improving the results even further. This graph will permit to lift planar curve processing algorithms onto surfaces for, e.g., vector graphics. Our depth data structure can represent sparse volumetric data such as surfaces extremely efficiently due to their inherent 2D property allowing us to detect changes in point clouds in real-time. Our work on sampling extends into 3D where all kinds of simulations, such as heat simulations, fracturing, or deformations can be speed up considerately while preserving the effects on small details.