Attention-Enhanced U-Net Segmentation for Reliable Detection of Circulating Tumor-Associated Cells

Abstract Background Circulating tumor associated cell (CTAC) detection-based multi-cancer early detection (MCED) strategies may be hindered by the rarity of CTACs among millions of peripheral blood nucleated cells (PBNCs). We developed an advanced U-Net-based encoder-decoder model for pixel-level CTAC discrimination that integrates attention-gated skip connections to preserve morphological and fluorescence details. Methods Model suitability was explored in an initial cohort of asymptomatic individuals (n = 428) and patients with advanced solid tumors (n = 354). A case-control study assessed clinical performance in therapy-naive stage I/II cancer patients (n = 185), individuals with benign conditions (n = 129), and asymptomatic individuals (n = 111). The model was then validated across four prospective studies on distinct populations: recurrent cancer cases with low tumor burden (n = 224); patients with solid tumors in the peri-operative setting (n = 17); suspected cancer cases (n = 259); and asymptomatic individuals (n = 7,183), respectively. All studies used blinded peripheral blood specimens from which PBNCs were isolated, stained for EpCAM / Hoechst 33342, and imaged. Ground truth annotations were established via pathologist review. The U-Net pipeline encoded spatial information in the images via convolutional and pooling layers and generated pixel-wise segmentation masks to identify CTACs. In all studies, sensitivity was based on CTAC detection rate in cancer specimens and CTAC undetectability rate in specimens from healthy asymptomatic individuals or those with benign conditions Results In the exploratory study, the model had 90.68% (95% CI: 87.16%, 93.50%) sensitivity and 99.53% (95% CI: 98.32%, 99.94%) specificity. In the case-control cohort, the model had 88.65% sensitivity (95% CI: 83.17%, 92.83%), 78.95% (95% CI: 71.03%, 85.53%) specificity in benign conditions, and >99.9% specificity in asymptomatic individuals. Among the four prospective studies, the model had: (a) 91.96% (95% CI: 87.60%, 95.17%) sensitivity in pretreated patients with low tumor burden; (b) 100% sensitivity in pre-surgery specimens, and 29.41% sensitivity in post-surgery specimens; (c) 96.34% PPV (95% CI: 93.22%, 98.05%) and a 32.35% NPV (95% CI: 25.58%, 39.95%) for diagnostic triaging; and, (d)11% PPV (95% CI: 31.72%, 53.24%) and 99.97% NPV (95% CI: 99.90%, 99.99%) for MCED in healthy asymptomatic individuals. Conclusions The attention-enhanced U-Net achieved robust, generalizable performance for CTAC-detection in case-control and prospective cohorts, supporting its clinical utility for accurate cancer detection.