GAN-based realistic phantoms for a multidomain approach in surgical action detection

Surgical training phantoms can be valuable tools not only for medical training but also in the development of technological innovations. From these, artificial intelligence (AI) based applications are some of the most promising. However, the lack of extensive labeled datasets poses a big limitation in their development. To address this constraint, new approaches have been explored, such as leveraging videos from surgical training phantoms as a source of cross-domain knowledge. However, these also present some limitations, including a lack of realistic traits such as blood and bleeding. We propose the implementation of a Generative adversarial network (GAN)-based image-to-image (I2I) translation method to provide realism to surgical training phantom images potentially enhancing these phantoms to function as more precise and customized training tools and increasing their utility for the development of different AI applications. <br/><br/>The cycle-GAN model is employed to successfully translate images from the intra-operatory domain onto the phantom domain. Data were extracted from the from the SARAS-Mesad challenge, which includes both real surgeries and phantom images from laparoscopic prostatectomy procedures. An action detection task is then used to evaluate the quality of our generated realistic phantom images. <br/><br/> Our findings show that using the combination of realistic phantom images surpasses the use of non-realistic phantom images (meanAP 17,0 and 16,6, respectively), suggesting that the generation of realistic synthetic data can be achieved with good results and has the potential to improve the performance of deep learning-based action detection in the context of Minimally Invasive Surgery (MIS), offering a promising path for future developments in intelligent surgical assistants.