Surgical RARP copilot: a vision language model for robot-assisted radical prostatectomy

Complex surgical procedures may benefit from AI systems that integrate visual and textual data for holistic scene understanding. We present Surgical RARP Copilot, a vision-language model for robot-assisted radical prostatectomy (RARP) that enables open question answering during surgery. We adapted a large language model to RARP literature and used it to generate a dataset of RARP images paired with ~1 million Q&A examples to train the model. Performance was evaluated for open-domain Q&A, surgical phase recognition, and instrument detection, and the system was deployed and tested in real time during a live operation—the first surgical VLM implemented in live robotic surgery. On unseen RARP procedures, Copilot showed robust performance across tasks. This work demonstrates feasible real-time AI guidance and suggests benefits for training, team communication, and knowledge support; future work includes broadening procedures and measuring clinical impact of such a system.