Search for a command to run...
When respondents do not understand the meaning of a survey question, they will not supply valid and reliable answers. Survey methodologists should therefore benefit from computer tools and other analytical schemes that help them identify problems with questions with respect to comprehension difficulty. We developed a Web facility called Question Understanding Aid (QUAID; www.psyc.memphis.edu/quaid.html) that assists survey methodologists in identifying problems with the wording, syntax, and semantics of questions on questionnaires. The survey methodologist enters the question into the Web facility, along with any context information and answer alternatives that accompany the question. QUAID quickly returns a list of potential problems with question comprehension, including unfamiliar technical terms, vague or imprecise relative terms, vague or ambiguous noun phrases, complex syntax, and working memory overload. This article describes QUAID and some empirical studies that have assessed the validity and utility of QUAID’s critiques of questions. The output of QUAID was compared with the judgments of experts in language, discourse, and cognition during the development of the tool. In one evaluation, expert survey methodologists critiqued and revised problematic questions, whereas in a second evaluation survey methodologists evaluated the quality of original problematic questions, questions revised with the assistance of QUAID, and questions revised without QUAID. In a third evaluation, eye-tracking data were collected while respondents read questions on a computer screen and answered questions aloud. Respondents had a tendency to give up processing difficult questions too early (called an early exit), which potentially threatens the validity of the respondents’ answers. Survey methodologists are encouraged to use QUAID and further evaluate its validity and utility.