Decoding Emotions: Facial Expression Recognition and Semantic Rating Prediction with Deep Learning

This work comprises of three studies. We use the Japanese Female Facial Expression (JAFFE) dataset, comprising of 213 images of 10 Japanese female subjects, depicting six primary emotions: Happy, Sad, Surprised, Angry, Disgust, and Fear, to conduct three different studies. In the first study, we developed a Convolutional Neural Network (CNN) model to classify major human facial expressions using only the image data. To facilitate this, we constructed a dataset in CSV format containing two columns: the image file names and t heir corresponding expressions, which were encoded based on the naming convention of the dataset. Our model achieved a testing accuracy of 97.08%, demonstrating its effectiveness in recognizing facial expressions. In our second and third studies, we focused on predicting individual ratings for the expressions using the semantic ratings provided in the accompanying text file. These ratings were derived from psychological evaluations on a 5-point scale for the six emotions. We developed a multi-class regression CNN model to quantify these ratings. The second study included all six expressions and yielded a testing accuracy of 75.82%, while the third study, which excluded the expression of Fear, achieved a higher testing accuracy of 86.06%. These findings suggest that Fear may be processed differently from the other basic facial expressions, consistent with observations in psychological studies. This work highlights the potential of using deep learning techniques to quantify and understand the subtle distinctions of human facial expressions.