Enhancing Large-Scale Consumer Prediction Models Using Generative AI Dataset Expansion and Probabilistic Output Mechanisms

This study presents advancements in large-scale consumer prediction models by leveraging generative AI techniques. Building upon previous research that utilized a subset of 11,000 respondents from a 150,000-participant dataset, we now incorporate the entire dataset to enhance model generalization. Additionally, we introduce a probabilistic output mechanism, enabling the model to generate probability distributions over multiple-choice responses rather than deterministic outputs. These enhancements have led to a significant improvement in prediction accuracy, with the correlation coefficient increasing from 0.68 to 0.80 and the root mean square error (RMSE) decreasing from 0.18 to 0.06. These findings underscore the potential of integrating comprehensive datasets and probabilistic outputs in generative AI models for more reliable consumer behavior predictions.