RAD-OAEnsemble: Residual attention DenseNet with radiograph-EHR fusion and dual-stream deep feature embedding for knee osteoarthritis risk prediction

Knee osteoarthritis (OA) is a popular and progressive degenerate joint disease and require a predictor of risk at the primary stage to predict drug treatment and propose an early intervention. The current methods of deep learning are not sufficient enough to deal with integrating the heterogeneous modality component, modeling of irregular temporal follow ups, and capture of the fine grained variations of pathological types. In order to address such limitations, this paper suggests a new multimodal approach named, RAD-OAEnsemble which introduces three main innovations. First, a novel Cross Bi-modal Heterogeneous Graph Fusion Network (CBHGF-Net) is introduced for the multi-scale semantic fusion of Radiographs and Electronic Health Records (EHRs). Second, a Modified DenseNet with Multi-scale Attention Criss-Cross Blocks (MACC-Blocks) is presented to extract the anatomical features of knee OA patterns by utilizing dilated convolution, criss-cross attention, residual attention gating. Third, a Dual-Stream Tranfo-Convo (DTC) Deep Temporal Embedding module is proposed that consists of a Time-Warped Convolutional Blocks and a Temporal Decay Encoded Informer for the knee OA risk prediction. The suggested approach delivered better outcomes with accuracy of 99.275%, precision of 98.175%, specificity of 99.165%, sensitivity of 98.95%, and F1-score of 98.175% in having considerably a low rate of false positives (1.875%) and false negatives (2.175%). The findings support the relevance of the suggested framework in providing resilient, time-sensitive, and explainable OA progression predictions by multimodal combination and high-fidelity feature learning.