Mixed reality in surgical telepresence: a novel extended mean value cloning with automatic trimap generation and accurate alpha matting for visualization

Abstract The aim of this research is to propose an extended mean value cloning algorithm with automatic trimap generation and accurate alpha matting. This implementation improves the visualization accuracy of the merged video by reducing the discolored and smudging artefacts of the remote surgeon’s boundary. It also makes the merge robust for the illumination changes by taking less processing time in real time surgery. The proposed system uses automatic trimap generation from the source video for accurate foreground extraction. Extended mean value cloning with gradient mixing is then applied for the cloning with optimized alpha matting for accurate and realistic video composition. The proposed system improved the visualization accuracy by providing almost 99.7% visibility of the pixels compared to the state-of-the-art solution, which provides 99.1% visibility of pixels. The overlay error was reduced from 0.93 mm to 0.63 mm. The processing time was also reduced. The proposed solution processed 8 frames per second, which is less time than the state-of-the-art solution, which processed 5 frames per second. The extended mean value cloning smooths the differences that presented in the target and source frames for seamless and realistic blending of pixels. The automatic trimap generation reduced the risk of false foreground selection and the generated optimal trimaps improved the alpha matte quality, which is optimized to reduce the smudging artefacts completely and to produce accurate visualization of the final merged image.