Leaping to conclusions

Publisher Website
Google Scholar
Abstract
Dangers of not finishingIain Chalmers, an editor of the James Lind Library and a previous director of the UK Cochrane Centre, says: “The issue is basically about the play of chance. You can get big swings by chance which may be statistically significant in both directions.” He explains how preliminary research is vitally important in interim analyses. “ISIS I was a trial of atenolol in acute myocardial infarction. One of the reasons this trial was done was that a systematic review had suggested that it would decrease risk of early death by 15%. On the basis of this promising information, it was decided to do a very large trial to get a confident handle on the size of effect. “The data monitoring committee found that early results suggested that atenolol was harmful, and statistically significantly so. But in assessing these interim results, it took into account that the evidence from the systematic review of all previous trials suggested that they might well be a reflection of the play of chance. So it allowed recruitment to the trial to continue.“By the time the trial had ended it had shown that atenolol reduced mortality by 15%, exactly as predicted by the systematic review of all previous trials. In other words, the systematic review protected the committee from making an inference which, in retrospect, would have been dangerous. The lesson is, ‘Don’t embark on a new trial before you know what happened in all previous trials.’ And this means all existing trials—not just those in print. If people are not publishing trials—for example, because of stopping them early due to adverse effects—then you will have a biased sample.”However, interim results may also overestimate the extent of harm of an intervention. Referring to the CRASH trial, which assessed the effects of steroids given to people with acute traumatic brain injury, Sir Iain says, “A systematic review had shown that this treatment might be helpful but that it was equally likely that it was harmful. This was the basis for doing the trial. The trial was stopped when an interim analysis showed a statistically significant harmful effect of the drug. Although the study provided strong evidence to justify abandoning this treatment, the actual size of the harmful effect was probably an overestimate, as the authors noted. In other words, both false and true harmful effects may be overestimated.”

This publication has 3 references indexed in Scilit: