Unlocking Emotions: A Novel Fusion of Facial Expressions and Pupillometry

The study investigates the enhancement of emotion detection through a multi-modal approach that integrates facial expressions and pupillometry. It evaluates two key fusion techniques: weighted probability fusion, which optimizes emotion recognition by prioritizing more accurate modalities, and feature fusion, which consolidates various data into a single model to learn complex patterns from both modalities. The study demonstrates that these multimodal methods noticeably outperform single-modality approaches; based on our analysis, a performance boost up to $10 \%$ was achievable under specific emotion classification scenarios using only $30 s$ of pupil data and over a database of 30 users. The work carried out here offers promising applications in user experience in various sectors, e.g. banking systems, where understanding customer emotions could improve service delivery.