Facilitating critical appraisal of evidence using automation tools

Abstract The European Food Safety Authority (EFSA) initiated the “Roadmap for actions on Artificial Intelligence for evidence management in risk assessment” project in May 2021, aiming to optimize risk assessment processes by integrating Artificial Intelligence (AI) methodologies. A key focus is the automation of the Risk of Bias (RoB) assessment, a critical and time‐consuming step in systematic reviews that evaluates the methodological quality of studies. Implemented under Framework Contract OC/EFSA/AMU/2021/03 ‐ Specific Contract 3, this project built and piloted a prototype automation tool to support the risk of bias assessment process. This report details the steps taken to define current EFSA processes, select appropriate critical appraisal tools and use cases, and develop and validate a prototype automation tool based on a GPT model with zero‐shot learning (AutoCAT). The implementation phase includes integrating AutoCAT with existing systematic review platforms, and deployment within the EFSA environment. The assessment phase involved evaluating the tool with development and system test datasets, and user testing with EFSA scientific experts. The project's outcomes are discussed, highlighting the limitation of the pilot prototype and proposing future work and potential applications in other contexts.