Assessing Inter-rater Agreement Across Five Teams Applying the Global Trigger Tool to Review 200 Inpatient Medical Records

Objective: A national quality indicator for hospital adverse events (AEs) could be based on Global Trigger Tool (GTT) data, with its validity depending on consistent reproducibility within the same GTT team and across independent GTT teams. A few studies have examined the inter-rater agreement between 2 independent teams’ ratings of the same medical records, but not across a larger number of teams, which was the objective of this study. Methods: Five trained and experienced GTT teams across Norway reviewed the same 200 randomly sampled records. The inter-rater agreement across the teams on the presence or absence of at least one AE was assessed, together with agreement on the number of AEs and AE severity level. Differences in the teams’ ratings were evaluated using a generalized linear mixed model, with adjustment for relevant covariates, and tested with a nonparametric method. Results: The 5 teams agreed on AE occurrence in 57.5% of the records, giving moderate agreement (Fleiss’ κ: 0.48; Gwet’s AC1: 0.65). Records with agreement on AE occurrence were clinically more complex, with longer LOS, more diagnoses and procedures, and higher readmission rates than those with disagreement. Teams 4 and 5 identified the highest number of records with AEs, and the differences in the 5 teams’ ratings were significant. Conclusions: We point to a need for strengthening reviewer calibration and training to obtain higher agreement across independent GTT teams. This is a prerequisite for applying GTT data in a national quality indicator for AEs in hospitals.