What makes marking reliable? Experiments with UK examinations

Marking reliability is purported to be produced by having an effective community of practice. No experimental research has been identified which attempts to verify empirically the aspects of a community of practice that have been observed to produce marking reliability. This research outlines what that community of practice might entail and presents two experimental studies on the effects of particular aspects of community of practice on examiners' marking reliability. In the first study, the impact of exemplar work is investigated: examiners were provided with mark schemes and some examiners were provided with exemplar scripts and given feedback about the marking of those scripts. In the second study, the effects of discussion of the mark scheme are explored: all examiners received mark schemes and exemplar scripts, but some examiners did not attend a coordination meeting. Neither procedure (use of exemplar scripts or discussion between examiners) demonstrated an improvement in marking reliability, which called into question the predictive utility of the theory of community of practice.