Issues in Measuring Reliability
- 1 July 1999
- journal article
- research article
- Published by SAGE Publications in Written Communication
- Vol. 16 (3) , 354-367
- https://doi.org/10.1177/0741088399016003004
Abstract
In many literacy studies, it is important to establish the reliability of independent observers' judgments. Reliability most commonly is measured either by the percentage of agreement or the correlation between the observers' judgments. This article argues that the percentage of agreement measure is more difficult to interpret than are correlation measures because of the following: (a) the effects of chance agreement are not accounted for automatically by the percentage of agreement measure; and (b) rates of chance agreement are strongly influenced by the variability of the data, by “ceiling” and “floor” effects, and by the scoring of near agreement as perfect agreement. For these reasons, the authors recommend that the field of literacy research adopt correlation as the standard method for estimating the reliability of observers' judgments.Keywords
This publication has 32 references indexed in Scilit:
- The Uses and Complexity of Argument Structures in Expert and Student Persuasive WritingWritten Communication, 1998
- Which is more Productive, Writing in Binge Patterns of Creative Illness or in Moderation?Written Communication, 1997
- Rhetoric in CompetitionWritten Communication, 1996
- The effects of strategy instruction on the comprehension performance of at‐risk studentsReading Research Quarterly, 1996
- Valuing differences: Portnet's first yearAssessing Writing, 1995
- Text Cues and Strategies Successful Readers Use to Construct the Gist of Lengthy Written ArgumentsReading Research Quarterly, 1995
- The Nature of Fourth Graders' Sociocognitive Conflicts in Peer-Led and Teacher-Led Discussions of LiteratureReading Research Quarterly, 1995
- Exploring Individual Differences among Novices Reading in a Specific Domain: The Case of LawReading Research Quarterly, 1995
- Toward the instructional utility of large-scale writing assessment: Validation of a new narrative rubricAssessing Writing, 1995
- The Emergence of GenresWritten Communication, 1994