A Cost–Benefit Analysis of Automatic Item Generation

Abstract Automatic item generation (AIG)—a means of leveraging technology to create large quantities of items—requires a minimum number of items to offset the sizable upfront investment (i.e., model development and technology deployment) in order to achieve cost savings. In this cost–benefit analysis, we estimated the cost of each step of AIG and manual item writing and applied cost—benefit formulas to calculate the number of items that would have to be produced before the upfront costs of AIG outweigh manual item writing costs in the context of K‐12 mathematics items. Results indicated that AIG is more cost‐effective than manual item writing when developing, at a minimum, 173 to 247 items within one fine‐grained content area (e.g., fourth‐ through seventh‐grade area of figures). The article concludes with a discussion of implications for test developers and the nonmonetary tradeoffs involved in AIG.