SSCDV: Social media document embedding with sentiment and topics for financial market forecasting

For reducing investment risks, predicting the volatility of financial markets is crucial. We propose a method for effectively embedding social media posts to facilitate accurate predictions of financial market trends. While discussions on social media inherently consist of paired information - a topic and its sentiment mood - most conventional studies have produced embeddings focusing only on either topic or sentiment information. This approach tends to neglect the intertwined nature of topic and sentiment, thereby overlooking potentially valuable information for market predictions. In this study, we overcome this challenge by introducing a novel document embedding technique that explicitly leverages both topics and sentiments collaboratively for market forecasting. The obtained embeddings are co-trained with financial time series data in a machine learning model, and their efficacy is evaluated through a market prediction task. Benchmarking against various existing document embedding techniques, our model demonstrated superior performance in terms of F-1 Score and Matthews correlation coefficient. The proposed model was further assessed from a practical viewpoint, utilizing investment simulations based on its predictions. These simulations confirmed the model’s potential to generate profits even during heightened market volatility, demonstrating its effectiveness as a real-world investment risk mitigation model. Model interpretation using SHapley Additive exPlanations revealed that while some topic-sentiment pairs on social media consistently contribute to market forecasting, others have only a transient impact. The SHapley Additive exPlanations experiment was also compared to the ablation model and showed that the proposed embedding allows for effective prediction by treating topics and sentiment jointly.