Estimating Interinterviewer Reliability for Interview Schedules used in Special Education Research

1 April 1991

journal article
research article
Published by SAGE Publications in The Journal of Special Education

Vol. 25 (1) , 73-89
https://doi.org/10.1177/002246699102500105

Abstract

Using data from an individually administered interview schedule, the Consumer Satisfaction Inventory (CSI), interinterviewer reliability was estimated with several different approaches: simple percentages of agreement, kappa and weighted kappa, Pearson correlations, t tests on interviewers' means, and generalizability (G) theory techniques. The reliability estimates varied, sometimes widely. Differences and similarities among the approaches are discussed, and some suggestions are given to aid researchers in choosing estimation techniques for particular situation.

Keywords

This publication has 23 references indexed in Scilit:

A Generalization of Cohen's Kappa Agreement Measure to Interval Measurement and Multiple Raters
Educational and Psychological Measurement, 1988
Integration and generalization of kappas for multiple raters.
Psychological Bulletin, 1980
BACK TO BASICS: PERCENTAGE AGREEMENT MEASURES ARE ADEQUATE, BUT THERE ARE EASIER WAYS
Journal of Applied Behavior Analysis, 1979
A GRAPHICAL JUDGMENTAL AID WHICH SUMMARIZES OBTAINED AND CHANCE RELIABILITY DATA AND HELPS ASSESS THE BELIEV ABILITY OF EXPERIMENTAL EFFECTS
Journal of Applied Behavior Analysis, 1979
Using generalizability theory in behavioral observation
Behavior Therapy, 1978
Observer Agreement and Reliabilities of Classroom Observational Measures
Review of Educational Research, 1978
Measuring nominal scale agreement among many raters.
Psychological Bulletin, 1971
Large sample standard errors of kappa and weighted kappa.
Psychological Bulletin, 1969
Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit.
Psychological Bulletin, 1968
A Coefficient of Agreement for Nominal Scales
Educational and Psychological Measurement, 1960