A model of calibration for subjective probability distributions and its application to aggregation of judgements

Abstract A simple model is proposed of the statistical structure underlying the calibration of auditors' subjective probability distributions for account balances, and potentially for other unknown quantities. The model relates calibration curve shape to two parameters which represent over‐ or underconfidence and over‐ or underestimation. It is fitted to data from expert auditors. Different types of account appear to have different calibration characteristics. The model helps predict approximately the effects on calibration of aggregating individual subject distributions. Aggregation improves accuracy, but produces a strong tendency towards underconfidence. One aggregation method, predicting the best judgement in the group and using it as the group judgement, is found to be quite effective, much better than averaging the fractiles of individual distributions.