Measurement of Observer Agreement
Top Cited Papers
- 1 August 2003
- journal article
- Published by Radiological Society of North America (RSNA) in Radiology
- Vol. 228 (2) , 303-308
- https://doi.org/10.1148/radiol.2282011860
Abstract
Statistical measures are described that are used in diagnostic imaging for expressing observer agreement in regard to categorical data. The measures are used to characterize the reliability of imaging methods and the reproducibility of disease classifications and, occasionally with great care, as the surrogate for accuracy. The review concentrates on the chance-corrected indices, kappa and weighted kappa. Examples from the imaging literature illustrate the method of calculation and the effects of both disease prevalence and the number of rating categories. Other measures of agreement that are used less frequently, including multiple-rater kappa, are referenced and described briefly.Keywords
This publication has 24 references indexed in Scilit:
- Breast imaging reporting and data system standardized mammography lexicon: observer variability in lesion description.American Journal of Roentgenology, 1996
- Modeling Approaches for the Analysis of Observer AgreementInvestigative Radiology, 1992
- Receiver Operator characteristic (ROC) Analysis without TruthMedical Decision Making, 1990
- High agreement but low kappa: II. Resolving the paradoxesJournal of Clinical Epidemiology, 1990
- High agreement but low Kappa: I. the problems of two paradoxesJournal of Clinical Epidemiology, 1990
- Indices of discrimination or diagnostic accuracy: Their ROCs and implied models.Psychological Bulletin, 1986
- The Effect of Verification on the Assessment of Imaging TechniquesInvestigative Radiology, 1983
- Improving Diagnostic Accuracy: A Comparison of Interactive and Delphi ConsultationsInvestigative Radiology, 1977