SEMINARS ON LIVER-DISEASE .9. STATISTICAL-ANALYSIS OF REVIEWER AGREEMENT AND BIAS IN EVALUATING MEDICAL ABSTRACTS

Google Scholar
  • 1 January 1976
    • journal article
    • research article
    • Vol. 49  (4) , 373-383
Abstract
Observer variability affects virtually all aspects of clinical medicine and investigation. One important aspect, not previously examined, is the selection of abstracts for presentation at national medical meetings. Abstracts (109) submitted to the American Association for the Study of Liver Disease, were evaluated by 3 "blind" reviewers for originality, design-execution, importance and overall scientific merit. Of the 77 abstracts rated for all parameters by all observers interobserver agreement ranged between 81-88%. However, corresponding intraclass correlations varied between 0.16 (approaching statistical significance) and 0.37 (P < 0.01). Specific tests of systematic differences in scoring revealed statistically significant levels of observer bias on most of the abstract components. Moreover, the mean differences in interobserver ratings were quite small compared to the standard deviations of these differences. These results emphasize the importance of evaluating the simple percentage of rater agreement within the broader context of observer variability and systematic bias.