Surgical Quality Assessment in Gynecologic Laparoscopy

In the domain of endoscopic surgery, the operating surgeons perform all actions based on images from the inside of the patient, produced by a tiny camera with a light source called the endoscope. Nowadays, these images are typically also recorded and stored in a video archive for later use. Reasons for this are manifold but a very important one is the post-hoc inspection of the video footage for assessing the technical quality of the surgical actions, also known as surgical quality assessment. Through retrospective video review, technical errors are identified and the surgeon is made aware of them, in order to avoid such errors in the future. It is known that this process of managing technical errors in surgery improves patient outcome and increase surgical quality. However, Currently, the video review is performed manually by an expert assessor, who uses a common video player, a checklist, and some external notes. The problem with this approach, however, is that it is very tedious, inefficient and error-prone, because no supporting software tools are available. In this research project we aim at improving this currently inconvenient process of surgical quality assessment. In a joint effort with multimedia experts (from Klagenfurt University) and medical experts (from Medical University of Vienna) we investigate fundamental research questions associated with surgical quality assessment. More precisely, we evaluate deep learning and video retrieval techniques for automatic detection of technical errors in laparoscopic surgery. Hypotheses: Our main hypothesis is that we can abstract and model the semantics of surgical quality assessment and improve (optimized/speed-up) the manual process through a combination of appropriate computing approaches. This combination includes machine learning and video retrieval methods, that can automatically learn and retrieve technical errors in the video footage and thereby support the medical expert at his/her work. Methods: Fundamental methods of this project are: multimedia information retrieval, video similarity search, machine learning, development of software prototypes, creating data sets of ground truth with manual annotation, as well as quantitative and qualitative studies. What is new and/or special about the project? Software tools to support the process of surgical quality assessment are currently not available, rendering the inspection process inconvenient and cumbersome. Therefore, many surgeons currently have no time for detailed inspection of the video footage. With this research project we are performing pioneering research work to find out how to improve the efficiency of surgical quality assessment. This will significantly impact medical care as it will allow clinicians to perform more video reviews for technical errors (and, hence, improve patient safety).

The SQUASH project investigated how video analysis with deep learning and video retrieval can support surgical quality assessment in gynecologic laparoscopy. Currently, surgical quality assessment relies heavily on clinicians manually reviewing recorded videos, which is time-consuming and inconvenient for larger datasets. Our research explored whether automatic recognition of relevant content semantics and the application of specific video content search could enhance this process by providing more structured and efficient analysis and search tools. To achieve this, we collaborated with clinicians to collect and annotate several datasets of gynecologic laparoscopy videos. Using deep learning techniques, we trained AI models to recognize relevant surgical elements, including instruments, anatomical structures, pathologies, surgical actions and events. We designed content recognition methods by applying convolutional neural networks (CNNs), recurrent neural networks (RNNs), and transformer-based models for the extraction of relevant semantics from recorded video data. In addition to recognition, we developed interactive video search tools that allow users to explore large video archives efficiently. These tools enable content-based search, where specific actions or situations can be retrieved using text, object, or action filters. Our participation in international image and video search competitions demonstrated the effectiveness of these methods, and allowed for comparison with other research colleagues. The findings of this project confirm that deep learning can greatly assist in analyzing surgical videos. Interactive video exploration tools that integrate recognized semantics allow to improve and optimize the time-consuming process of surgical quality assessment via content and similarity search. We have publicly released parts of our dataset to facilitate further studies and improve the transparency of AI-driven medical analysis.