Reading comprehension: a meta-analysis comparing standardized and non-standardized assessment results

The primary purpose of this meta-analysis was to determine how reading comprehension results differed across distal and proximal assessment measures for vocabulary, content, and strategy instruction. A total of 1557 studies were screened from the Education Source database, the ERIC database, and three previous meta-analyses on the topic: Hwang, Cabell and Joyner (Sci Stud Read, 26(3): 223–249, 2021), Filderman, Austin, Boucher, O’Donnell and Swanson (Exceptional Children; 88(2): 163-184, 2022), and Silverman, Johnson, Keane and Khanna (Read Res Quarterly; 55(Suppl 1): S207–S233, 2020). In order to be eligible, studies had to meet the following criteria: assessed a reading comprehension pedagogy, included a comprehension measure, had at least 20 participants, and provided sufficient data for calculating a pooled standard error and Cohen’s d. This resulted in a meta-analysis of 71 studies and 115 effect sizes were calculated, with effect sizes weighted by inverse variance to reflect sample size proportionality. On standardized tests, the mean effect size for vocabulary was 0.22, for content was 0.11, for cognitive strategies was 0.20, for meta cognition strategies was 0.17. However, effect sizes were most statistically significant when vocabulary, and cognitive strategy instruction were combined (d = 0.23). All common comprehension pedagogies showed wide heterogeneity. The effectiveness of the pedagogy was entirely context dependent. There was no one pedagogy that was efficacious in all contexts.