A Novel EEG–ECG Based Fusion Model for Multimodal Emotional Classification

The identification of emotional states through physiological signals is vital in human–computer interaction, mental health monitoring, and management. In contrast to existing models, which use either a combination of visual and electroencephalogram (EEG) features or individual features, this study introduces a hybrid multimodal machine learning framework that fuses EEG and electrocardiogram (ECG) features for emotion classification. The EEG indeed captures the brain’s neural activity associated with emotional states, while the ECG monitors the heart’s electrical signals, reflecting autonomic responses that are highly responsive to emotional fluctuations. Integrating these physiological signals enhances the representation of emotional states by capturing complementary information from both the cortical and autonomic nervous systems. To exploit this observation, we explore the combination of a temporal convolutional network (TCN) and a long short-term memory (LSTM) network. This work considers four distinct emotion levels based on valence and arousal dimensions. Experiments on two benchmark datasets, namely, DREAMER and Multimodal, demonstrate the superiority of the proposed method over state-of-the-art methods.