The Reliability of Peer Reviews of Papers on Information Systems
- 1 February 2004
- journal article
- other
- Published by SAGE Publications in Journal of Information Science
- Vol. 30 (1) , 2-11
- https://doi.org/10.1177/0165551504041673
Abstract
This paper analyses the reliability of the double-blind peer review systems used for submissions to the 2001 and 2002 UK Academy for Information Systems (UKAIS) conferences. The level of reliability found in the first conference was marginally lower than would be expected from a model based on chance. In the second conference the reliability level was significantly better, but still low. The paper explores some of the implications of this for the reviewing system, and suggests a model for assessing the impact of low levels of reliability.Keywords
This publication has 17 references indexed in Scilit:
- A multifaceted portrait of a library and information science journal: the case of the Journal of Information ScienceJournal of Information Science, 2003
- Something rotten at the core of science?Trends in Pharmacological Sciences, 2001
- Reproducibility of peer review in clinical neuroscience: Is agreement between reviewers any greater than would be expected by chance alone?Brain, 2000
- Development of the Review Quality Instrument (RQI) for Assessing Peer Reviews of ManuscriptsJournal of Clinical Epidemiology, 1999
- Critically evaluating an applications vs theory framework for research qualityOmega, 1999
- How reliable is peer review? An examination of operating grant proposals simultaneously submitted to two similar peer review systemsJournal of Clinical Epidemiology, 1997
- Report of the Ad Hoc Committee on Double-Blind RefereeingStatistical Science, 1993
- But the reviewers are making different criticisms of my paper! Diversity and uniqueness in reviewer comments.American Psychologist, 1990
- Peer review and theCurrent AnthropologyexperienceBehavioral and Brain Sciences, 1982
- Peer-review practices of psychological journals: The fate of published articles, submitted againBehavioral and Brain Sciences, 1982