Multimodal model integrating ultrasound and demographic data for the diagnosis of knee osteoarthritis

Ultrasonography (US) is useful for soft tissue delineation, but its diagnostic objectivity is limited by surgeon dependency and lack of guidelines. Diagnostic support based on deep learning may overcome these issues, and integrating demographic factors (age, sex, and body mass index [BMI]) may improve accuracy. However, multimodal convolutional neural network (CNN) models incorporating these factors have not been fully investigated for knee osteoarthritis (OA). In this study, we developed such a model and evaluated its usefulness for diagnosis and severity classification. We developed image-only and multimodal CNN models using 11 architectures pre-trained on ImageNet. OA severity was classified based on Kellgren–Lawrence grades. Diagnostic performance was assessed using the area under the curve (AUC), sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and F1 score. Among 491 limbs, 318 were diagnosed with OA. For diagnosing knee OA, a statistically significant improvement in performance of the multimodal model compared with the image-only model was observed only for AlexNet (AUC: 0.87 to 0.89, p = 0.036). No significant differences were observed between the models for other architectures, with differences in sensitivity, specificity, and F1 score remaining within 1–3%. SHapley Additive exPlanations analysis indicated that the contributions of age, sex, and BMI were generally small, and model predictions were predominantly driven by imaging features. In the stepwise evaluation, ResNet50 (image-only model) showed the highest sensitivity and NPV in the initial screening step (Step 1; 80% and 68%, respectively). For supplemental re-evaluation, ResNet152 and GoogLeNet demonstrated high specificity and PPV (96% and 97%, respectively). In severity classification among OA-positive cases (Step 2), VGG16 achieved the highest sensitivity and NPV (94% and 96%), whereas ResNet50 showed relatively high specificity and PPV (89% and 79%) for definitive diagnostic support following severity classification. CNN models using only US images demonstrated high diagnostic performance for knee OA. The addition of background factors provided limited benefit, suggesting that image-only models are sufficient for diagnostic support. This study was registered with the UMIN Clinical Trials Registration System: https://center6.umin.ac.jp/cgi-open-bin/ctr/ctr_view.cgi?recptno=R000056245; Trial Registration Number: UMIN000049395; Registration date: 2022/11/01.