Virtual and Augmented Reality Module for 3D Reconstruction

This interdisciplinary research project between computer science and medicine targets the development of a comprehensive image-guided tool for head and neck surgery with the main focus on mandibular fractures and mid-facial fractures. The tool supports the physicians in all treatment stages (diagnosis, planning, intervention and monitoring) and can also be used during clinical traineeship. A key point is the development of an algorithm for the semi-automatic and automatic segmentation of bone structures and soft tissue in CT and MRI acquisitions. A segmentation enables a three-dimensional localization, quantification and visualization of biological structures in a very short time. Based on the radiological images from the clinical routine, treatment diagnosis can be visualized, and surgical planning processes or alternative surgery options can be simulated. In addition, the postoperative simulation results can be presented preoperatively in a photo-realistic manner. In particular, this will be applied in the reposition of bony structures, the reconstruction of facial defects and the removal of tumors in complex anatomical areas. Moreover, we plan the development of patient-individual three-dimensional implants, which will 3D printed in- house for comparison with the external manufactured ones and for clinical usage. Furthermore, the medical personnel can be supported in real-time during surgery by an interactive navigation, where Augmented Reality integrates computer-generated objects into the operation field. The application reduces surgical complications and ensures a successful treatment result by pre- and intraoperative simulations. Hence, the number of necessary corrective surgeries and the operation time needed can be reduced, and a higher patient survival rate can be achieved. In addition, the combination with Virtual Reality glasses allows the virtual training of operations during medical education. The objective of the project is the establishment of an open source tool as basis for further research and developments for the participating universities. Currently, comparable tools are protected by strict licensing conditions and not accessible without restrictions for academic research. Existing software for clinical practice is not functionally stable enough and has too high error rates. Additionally, the usage imposes high financial costs, which makes a widespread application not possible. Finally, Augmented Reality in the clinical routine is an important topic for current surgical research and cannot be ignored by modern medical universities.

In this interdisciplinary research project between computer science and medicine, image-supported tools for head and neck surgery were developed, which support physicians in all medical phases (diagnosis, operation planning, intervention and control), but can also be used in teaching. One aspect of the project was the development of algorithms for the semi-automatic and automatic segmentation of bony structures and soft tissue based on CT and MRI data sets. The segmentation enables the three-dimensional localization, quantification and visualization of anatomical structures in a short time. Based on radiological image data from the clinical routine, treatment plans can be quickly visualized, and surgical planning processes or operative treatment options can be simulated. This is mainly used in the repositioning of bony structures, in the reconstruction of facial defects and in the removal of tumors in areas with complex anatomy. Furthermore, an augmented reality (AR) tool was developed with which the medical staff can be supported in real time by interactive visualization and navigation, even during an intervention. Using AR, virtual, computer-generated objects can be integrated into the real, physical surroundings. With the developed tool, anatomical structures and planning information are visualized directly on the patient, pre- or intra-operatively, and surgeons are guided interactively to the target anatomy. This can reduce surgical complications and ensure a successful treatment outcome. Consequently, the number of necessary corrective operations and the required operating time can be reduced. As part of the project, open-source applications were created as a basis for the further development at the participating universities. Similar applications are currently protected by strict licenses and are not fully accessible to academic research.