Automated retinal segmentation of hyperspectral images using a modified U‑Net with spectral multihead attention

Hyperspectral imaging (HSI) can capture spatial and spectral data of the retina. Retinal vessel segmentation enables quantitative analysis and assessment of retinal health. In this work, we developed an HSI system for mouse retina and a deep learning architecture to segment retinal vessels. Retinal images were manually segmented to establish ground truth. We proposed the use of multihead attention block for the spectral data, as well as Tversky loss for segmentation refinement. The model’s performance was assessed using Dice similarity coefficient (DSC) and the Jaccard (IoU) metrics. A comparative analysis was conducted against the manual segmentation, and five other segmentation methods: UNet, UNet++, UNet3+, segment anything model (SAM), and SAM2. We demonstrated that our spectral network achieved an IoU and DSC scores of 0.80±0.04 and 0.89±0.02, respectively. Our network outperforms other networks. The automatic segmentation method provides a tool for retinal analysis and quantification.