A Quantitative Study of Bias in Systematic Reviews

Publisher Website
Google Scholar
Abstract
The objective of this study was to identify potential sources of bias and quantify the size and direction of bias in systematic reviews and randomized placebo-controlled trials with low level laser therapy (LLLT) for chronic tendon and joint disorders. Literature search and statistical analysis of unreliable data (i.e. data that have been interpreted differently by different authors) was used to determine what direction these data were interpreted by reviewers. Seven systematic reviews and 36 randomized placebo-controlled trials were identified in the literature search. There was no significant correlation between PEDro method scores and outcome, and no publication bias existed when the total material of trials was considered. Inclusion of trials occurred in 69 cases, although trials fulfilled inclusion criteria in 105 cases. From 76 cases of comparable methodological assessment scores, intra-reviewer reliability was moderate to poor [intraclass correlation coefficient (ICC3,1): 0.47] across reviews. Exclusion of trials by method scores was contradicted by other reviews in 11/12 controllable cases. Two-thirds of the available trials were excluded from the review conclusions. A quarter of the trials that formed the final review conclusion had outcome results that had been interpreted as both positive and negative by different reviewers. Reviewers' selections in areas of identified disagreement were significantly in favour of their own final conclusions (p<0.002). Unreliable lack of trial inclusion, unreliable methodological assessments, erroneous interpretation of trial data and unreliable dichotomization of equivocal trial data were identified as sources of bias. One-third of the reviewers' selections at each successive reviewing stage were based on unreliable data, and biased systematically to support review conclusions. As publication bias was absent in this material, removal of methodological assessments would have reduced bias by one-third without compromising review validity.