Robustness of deep learning methods for ocular fundus segmentation: Evaluation of blur sensitivity

Abstract This article analyzes the sensitivity of deep learning methods for ocular fundus segmentation. We use an empirical methodology based on non‐adversarial perturbed datasets. The research is motivated by mass screening and self‐administered tests in which AI methods are needed and may be given images with focus issues. These substandard pictures are simulated using blurring algorithms of varying designs and kernel sizes which are subjected to a test of inter‐network and inter‐dataset sensitivity using Gaussian noise as a control. We test for simulated defocusing, motion blur, and Gaussian noise. The DRIVE, CHASE, and STARE datasets were used to generate 441 synthetic datasets for testing. Analysis of the resultant n = 37,888 sample has identified anomalies illuminating failure modes of state‐of‐the‐art ocular segmentation models. Data analysis led us to conclude that accuracy is a poor measure of sensitivity to input change and that even moderate levels of blur have dramatic effects on said sensitivity. We further concluded that architectures show larger variations in sensitivity to blur which do not match either their reported or measured performance on unblurred test datasets. Our analysis has attributed at least a part of the problem to overfitting non‐essential input dataset features and resolution sensitivity.