Inconsistent Responding on a Mixed-Worded Scale in the PISA 2022 Questionnaire: Prevalence and Predictors Across Seven Countries

Mixed-worded questionnaire scales, containing both positively and negatively phrased items, are intended to encourage thoughtful responses, but may result in inconsistent responses and wording effects and can reduce reliability and validity. This study examined the prevalence and predictors of inconsistent responding to a mixed-worded scale in the 2022 Programme for International Student Assessment (PISA) across seven countries. Using factor mixture analysis, findings revealed a low but variable rate of inconsistency (2–8%) across countries. Separate confirmatory factor analysis models on the effective and on filtered samples of consistent respondents showed improved fit and reliability indices for the filtered samples for all countries, suggesting that inconsistent respondents had unexpected response patterns. In logistic regression models, inconsistent responding was associated with lower PISA reading scores and often with reduced self-reported effort, suggesting that inconsistent responses may stem from lower reading ability and/or carelessness. The results contribute to ongoing discussions about the use of mixed wording in questionnaires and highlight the need to account for respondent behavior when analyzing survey data.