Systematic inconsistency in human and machine scoring of indices of malocclusion: the “Lazy-S Curve”

ABSTRACT Objectives To demonstrate systematic patterns relating “subjective” and “objective” measurement when scoring the American Board of Orthodontics Discrepancy Index (DI) and Objective Grading System (OGS) and Peer Assessment Rating (PAR) indices. Materials and Methods Digitized records of study models from 108 subjects scored by three trained raters were compared with scores from a proprietary digital scanning system with a built-in algorithm for producing these indices. Results Repeat measures of consistency using the digital system were perfect. Interrater consistency was typically moderately high (median ICC = 0.866). Correlations between human and digital measurement methods for the 33 component and index scores ranged from median intraclass correlation coefficients of 0.826 for DI, 0.568 for OGS, and 0.775 for PAR pretreatment and 0.636 for post-treatment PAR. The relationship between subjective and objective assessments showed common relationships: slopes significantly less than 1.00, and overestimation of small values and underestimation of large values. This pattern, known as the “lazy-S” curve, is characteristic of orthodontic and general measurement systems. The consistency for overjet and overbite indices was better between human and digital systems than across the two indices proportion to measure the same features. Conclusions The inconsistency between human and machine measurements of occlusal characteristics does not necessarily reflect inadequacy of either approach and may be interpreted instead as reflecting features of judgment orthodontists use to focus on the clinically most relevant data.