An Analysis of Sequential Patterns in Datasets for Evaluation of Sequential Recommendations

Sequential recommender systems are an important and in-demand area of research. These systems aim to use the order of interactions in a user’s history to predict future interactions. The premise is that the order of interactions and sequential patterns play an essential role. Therefore, it is crucial to use datasets that exhibit a sequential structure to evaluate sequential recommenders properly. We apply several methods based on random shuffling of user interaction sequences to assess the strength of sequential structure across 19 datasets, most of which are frequently used for evaluating sequential recommenders in recent top-tier conference papers. Since shuffling explicitly breaks sequential dependencies, we estimate their strength by comparing metrics for shuffled and original versions of the dataset. We propose three approaches for this assessment: a sequential rule method and two more advanced model-based approaches. The proposed methods are easily reproducible and consistent with each other. We also examine the impact of common preprocessing and postprocessing techniques, showing that they may affect conclusions about the presence of sequential structure. Finally, we introduce a method for distinguishing between recency-based and order-sensitive sequential patterns and for assessing their complexity. Our findings show that several popular datasets exhibit rather weak sequential structure.