A Multimodal Generative AI Framework for Realistic Interview Simulation and Comprehensive Behavioral Analysis

Abstract—The recruitment process is high-stakes, yet many candidates struggle due to anxiety and limited, unstructured feedback. Existing mock interview tools do not scale well and rarely capture behavioral signals beyond text. In this work, we present Interview Ai, a multi-modal system for interview simulation and performance analysis. The interaction is delivered through a conversational avatar that speaks and responds in real time, creating a more realistic setting than text-only systems. We model the interview flow as a finite state machine, which allows controlled transitions and adaptive follow-up questions. Candidate responses are evaluated using retrieval-based scoring, where answers are compared against exemplar knowledge rather than relying purely on model generation. The evaluation also incorporates measurable signals, such as filler word frequency, pauses, and response structure. In a study with 50 participants, users showed improvements in clarity, confidence indicators, and reduced hesitation compared to traditional practice. These results suggest that combining struc- tured control, grounded evaluation, and multi-modal interaction can provide effective and scalable interview coaching without replacing human decision-making. Index Terms—Large Language Models, LangGraph, Auto- mated Interview Systems, Computer Vision, Natural Language Processing, Speech Synthesis, GPT-4o-mini