Contrastive Multimodal Graph Vision Transformers for Context-Aware Visual Representation and Relational Reasoning

Vision transformers have recently advanced fast vision recognition operations, yet have low capacity to capture relational dependency and contextual semantics which impedes higher level reasoning. In the quest to overcome these shortcomings, this paper introduces CMG-ViT (Contrastive Multimodal Graph Vision Transformer) - a single framework that integrates the transformer-based learning of global features, graph-based relational reasoning, and contrastive multimodal alignment to the context-based visual understanding. This is done by using the input images and encoding them into patch-level representations and constructing a scene graph whereby objects, properties and binary relationships are represented. The heterogeneous representations are then added with the assistance of a graphconscience self-attention model, and it is fed into the transformer with topological priors to become superior models of relational features. Two view contrastive learning is a form of learning that combines visual and graph representations to a shared latent representation to allow semantic consistency source. Moreover, topology is held together with a regularization term, which provides the structural smoothness, which guarantees the integrity of relational graph of the latent representation. Massive analyses on several benchmarks such as Visual Genome, GQA, RefCOCOg, and HICO-DET, show that CMG-ViT is much higher than the state-of-the-art baselines, with significant improvements in recall, accuracy, and mean average precision. The findings indicate that adding graph reasoning and contrastive alignment to transformer systems is effective and the future of multimodal vision systems with the ability to perform deep semantic perception, situational reasoning, and interpretable visual inference is within reach.