Estimating Individual Rater Reliabilities

Publisher Website
Google Scholar
Abstract
Rating scales have no inherent reliability that is independent of the observers who use them. The often reported interrater reliability is an average of perhaps quite different individual rater reliabilities. It is possible to separate out the individual rater reliabilities given a number of independent raters who observe the same sample of ratees. Under cer tain assumptions, an external measure can replace one of the raters, and individual reliabilities of two independent raters can be estimated. In a somewhat similar fashion, estimates of treatment effects present in ratings by two independent raters can provide the external frame of reference against which dif ferences in their individual reliabilities can be evaluated. Models for estimating individual rater reliabilities are provided for use in selecting, evalu ating, and training participants in clinical research.