A More Powerful Method for Testing for Agreement between a Judge and a known Standard

Publisher Website
Google Scholar
Abstract
We assume that a judge's task is to categorize each of N subjects into one of r known classes. The design of primary interest is employed if the judge is presented with s groups, each containing r subjects, such that each group of size r consists of exactly one subject of each of the r types. The probability distribution for the total number of correct choices is developed and used to test the null hypothesis that the judge is “guessing” in favor of the alternative that he or she is operating at a better than chance level. The power of the procedure is shown to be superior to two other procedures which appear in the literature.