Emotion Recognition in Conversation Based on the Fine-grained Multidimensional Emotion Representation Learning

Traditional emotion recognition in conversation (ERC) studies are usually designed to predict a fixed set of predetermined emotion categories. This limited supervision diminishes the expressive power of the data, resulting in failure to capture the complexity of human emotions in conversation. Learning from a well-designed fine-grained representation of emotions offers a promising alternative that utilizes a wider range of supervision. In this paper, the proposed Fine-grained Multidimensional Emotion Representation Learning (FMERL) framework integrates multitask learning and contrastive learning, and extends the emotion representation of valence, arousal and dominance (VAD) from the psychological field to both continuous and discrete forms. Firstly, the emotion features from text, audio, and visual modalities are extracted. Then, the multimodal features are fused by a transformer-based model. The multitask learning module consists of three networks: the valence network, arousal network, and dominance network, for learning the continuous fine-grained emotion representations from the fused multimodal features. The contrastive learning aligns fused multimodal features with the discrete fine-grained emotion representations derived through prompt engineering applied to a large language model. The transferable ability of contrastive learning enables FMERL to map the semantic information of emotion representation and fused multimodal features into a shared embedding space, thereby understanding their semantic relationships and enabling zero-shot learning for unseen emotion classes. Experimental results on the IEMOCAP and MELD datasets have shown that FMERL achieves state-of-the-art performance in emotion recognition and implements zero-shot learning.