A Systematic Approach to Evaluate the Use of Chatbots in Educational Contexts: Learning Gains, Engagements and Perceptions

Generative Artificial Intelligence (GenAI) chatbots have demonstrated exciting potential to deliver high-quality, personalized learning at scale. However, there remains a gap in understanding the comprehensive impact of chatbots on learning gains. In this paper, we aim to provide researchers and educators with a replicable framework for a structured approach to evaluating the impact of chatbots on student learning. The framework includes a longitudinal randomized control trial (RCT) as the main methodology, measuring learning gains, engagement, and perception through various data sources and analytical methods. To demonstrate the efficacy of this framework, we built a pedagogically customized chatbot, called the Socratic chatbot, and analyzed the learning impact of student-chatbot interaction. Using the K-means method to determine the effect size of the study, we analyzed the significance of changes in pre- and post-test scores, conversation messages, conversation topics, and feedback. Overall, with the proposed framework, this study provides holistic and comprehensive evidence that the Socratic chatbot can be an effective tool for enhancing student learning gains, engagement and perception.