SC-MGM: soft clustering and Markov-GNN modeling for spatiotemporal community detection in video scene graphs

We introduce SC-MGM, an innovative framework for spatiotemporal community recognition within video scene graphs. Our objective is to find cohesive communities of objects that endure and evolve over time inside a succession of video frames represented as scene graphs. SC-MGM incorporates three complementary elements: a Graph Neural Network (GNN) for encoding intra-frame relational structures, a differentiable soft clustering module for seamless and trainable community assignment, and a Markov transition module that represents the temporal evolution of community prototypes. This integrated architecture facilitates comprehensive optimization of structural precision and temporal coherence. We assess SC-MGM using two extensive datasets, Action Genome and VIDOR, and exhibit consistent enhancements over current baselines regarding modularity, clustering accuracy, and interpretability. Qualitative representations demonstrate that our model accurately represents fluid community trajectories in accordance with semantic dynamics. The suggested method offers a systematic and comprehensible basis for dynamic scene comprehension, with possible applications in action analysis, video condensation, and relational inference.