Can Examinees Use a Review Option to Obtain Positively Biased Ability Estimates on a Computerized Adaptive Test?

Publisher Website
Google Scholar
Abstract
Part of the controversy about allowing examinees to review and change answers to previous items on computerized adaptive tests (CATs) centers on a strategy for obtaining positively biased ability estimates attributed to Wainer (1993) in which examinees intentionally answer items incorrectly before review and to the best of their abilities upon review. Our results, based on both simulated and live testing data, showed that there were instances in which the Wainer strategy yielded inflated ability estimates as well as instances in which it yielded deflated ability estimates. The success of the strategy in inflating ability estimates depended on the ability estimation method used (maximum likelihood versus Bayesian), the examinee's true ability level, the standard error of the ability estimate, the examinee's ability to implement the strategy, and the type of decision made from the ability estimate. We discuss approaches to dealing with the Wainer strategy in operational CAT settings.