The reliability of the College of Intensive Care Medicine of Australia and New Zealand viva examination

Examinations used to credential trainees for independent specialist practice in intensive care medicine should be evaluated to demonstrate their levels of reliability and validity. This study aimed to quantify the reliability of the College of Intensive Care Medicine of Australia and New Zealand (CICM) second part viva examination and report the variation due to candidate proficiency, station difficulty, case specificity, and examiner stringency. This retrospective analysis of the CICM examination database used generalisability theory to determine the overall reliability coefficient of the viva examination. Decision studies were used to calculate the number of stations required for a reliability coefficient of >0.8. The CICM assesses intensive care trainees in Australia and New Zealand for independent specialist practice. De-identified candidate results from 2019 to 2023 totalled 376 viva examination attempts with individual examiner scores from 2774 viva stations. The mean viva station score was 5.6 (standard deviation: 1.47). The reliability coefficient for the second part eight-station viva examination was 0.61. Sources of variance were candidate proficiency of 15.4 %, station difficulty of 22.8 %, case specificity of 51.5 %, examiner stringency of 1.4 %, and other error of 8.8 %. Twenty viva stations would be required to achieve a reliability coefficient of >0.8. The CICM second part viva examination has a reliability coefficient below the generally accepted value for a high-stakes examination. Case specificity and station difficulty made the largest contribution to score variance. Increasing the number of viva stations and introduction of pass/fail standard setting would be effective methods for increasing examination reliability.