Use and Effect of Caution Indices in Detecting Aberrant Patterns of Standard-Setting Judgments

Publisher Website
Google Scholar
Abstract
This study explored the use of the modified caution index (MCI) to identify judges whose patterns of item judgments were aberrant when compared with the pattern produced by the group of judges as a whole. It is possible, although speculative, that such a pattern of judgments might be based on incomplete information, flawed reasoning, or inattention to a judge's standard-setting tasks. The study also examined the effects on test standards and examinees' passing rates when the recommended test standards of judges whose patterns of item recommendations appeared aberrant were eliminated. Screening judges with large MCIs was found to have little effect on the standards recommended for a criterion-referenced reading test. However, substantially lower standards were recommended for a criterion-referenced mathematics test when judges with large MCIs were screened. Effects on examinees' passing rates were similar to effects on recommended test standards. The practical implications of these procedures and results are discussed.