Enhancing Teaching and Learning with Conversational AI Agents: Pilot Testing and Impact Assessment in Real Classroom Settings

This paper presents the evaluation of the educational impact of two conversational AI applications, one designed to support student selfstudy (Study Buddy) and the other to assist teachers in instructional planning (Teacher Mate) within the context of the AI4EDU project.The evaluation was conducted during the second piloting phase across four European countries in real classroom settings. In Greece,the evaluation involved 148 participants, 3 teachers and 145 students. The evaluation protocol included training workshops, pre- andpost-intervention surveys, system interaction data analysis, and classroom-based implementation. Greek participants were assigned toexperimental and control groups for comparative analysis. During a multi-week period, students used Study Buddy during self-study bothat school and at home, while teachers integrated Teacher Mate in lesson planning and student monitoring tasks. Quantitative data frommotivation and engagement questionnaires and interaction logs were triangulated with qualitative data from semi-structured interviewsand classroom observations. The analysis provided insights into the impact of AI tools on users’ educational performance, engagementand motivation and helped identify possible drawbacks and key aspects of AI implementation that can be exploited to demonstrate theissues of ethics, transparency and inclusive use of AI in real educational settings. In general, participants from both groups expressedfavorable opinions of the AI-driven applications. The combination of our study’s empirical results emphasizes the need to align AI toolswith user needs, deliver customized training, and cultivate trust and openness of AI applications. These initiatives are critical for fosteringinclusive, stimulating, and impactful AI-supported educational settings.