The relation between treatment benefit and underlying risk in meta-analysis

Publisher Website
Google Scholar
Abstract
The data To demonstrate the statistical pitfalls in an analysis of underlying risk as a source of heterogeneity we use data from a meta-analysis of randomised trials to assess the effectiveness of endoscopic sclerotherapy in reducing mortality in patients with cirrhosis and oesophagogastric varices.8 Nineteen such trials comparing sclerotherapy with a control were reviewed; the data relevant to our discussion are shown in table 1. In the following, we use proportion of deaths (expressed as a percentage) on a log odds scale as the measure of “underlying risk” and the log odds ratio as the measure of treatment effect. However, the principles apply more generally to other measures of treatment effect, such as the risk ratio or mean difference for a quantitative outcome, and corresponding measures of “underlying risk.” There was substantial evidence of heterogeneity (P<0.0001) in the observed odds ratios across the 19 sclerotherapy trials, and therefore an investigation of whether underlying risk was part of the explanation for different observed treatment effects is scientifically relevant.2 View this table: In this window In a new window Table 1 Mortality results from 19 trials of sclerotherapy taken from a published meta-analysis8 Relating treatment effect to underlying risk: three conventional approaches (1) GRAPH OF TREATMENT EFFECT AGAINST PROPORTION OF EVENTS IN CONTROL GROUP A natural measure of underlying risk in a trial population is the observed proportion of events in the control group. Figure 1 shows a graph of odds ratio of death (log scale) against proportion of deaths in the control group (log odds scale) for the data from the sclerotherapy trials. Each trial on the graph is represented by a circle, the area of which indicates the study size. The graph includes the line of predicted values obtained from a weighted regression. The estimated slope is -0.61 (95% confidence interval -0.99 to -0.23), giving strong evidence of a negative association—that is, an increase in treatment benefit (lower odds ratio) with increasing proportion of events in the control group. The conclusion from this analysis would be that underlying risk is a significant source of heterogeneity. Furthermore, there is a temptation to use point T in the figure to define a cut off value of risk in the control group and conclude that the treatment is effective (odds ratio below 1) only in patients with an underlying risk higher than this value. View larger version: In this window In a new window Fig 1 Treatment effect versus percentage of events in control group for sclerotherapy trials. The area of each circle is inversely proportional to the variance of the estimated treatment effect in the trial The problem with this interpretation is due to regression towards the mean.9 10 Because the outcome in the control group is being related to the treatment effect, an expression that also includes the control group outcome, a relation is expected. In the case where the treatment reduces the risk a high observed proportion of events in the control group will tend to lead to a larger observed treatment effect—and the converse when the observed proportion is low. In other words, the bias will lead to the potentially incorrect inference that the treatment is most beneficial among high risk patients and least among low risk patients. The size of the bias can be surprisingly large. In the extreme case when the treated and control group outcomes are unrelated the expected correlation can be -0.71.11 Underlying risk may indeed be a source of heterogeneity, but such a graph and regression will misrepresent any true effect. (2) GRAPH OF TREATMENT EFFECT AGAINST AVERAGE PROPORTION OF EVENTS IN THE CONTROL AND TREATED GROUPS Figure 2 is a graph of odds ratio of death (log scale) against the average proportion of events in the control and treated groups (log odds scale) for the data from the sclerotherapy trials; there is a slight increase in treatment effect (reduction in odds ratio) as the average proportion increases, but the evidence is unconvincing: the slope of the fitted line is -0.16 (-0.73 to 0.42). Use of the average has led to a different conclusion: underlying risk is not a significant source of heterogeneity. View larger version: In this window In a new window Fig 2 Treatment effect versus average percentage events in control and treated groups for sclerotherapy trials Unfortunately the validity of the approach using the average relies on the assumption that the true treatment effect does not vary between trials12; departures from this assumption will lead to bias in the size and direction of any observed association. To take an extreme example, consider a set of very large trials (where errors of measurement are negligible) which have the same underlying risk (as measured by proportion of events in the control group) but some of which have larger treatment benefits than others. A graph of treatment effect against average proportion will show a positive relation, whereas in truth there is no relation with underlying risk. Furthermore, there is also a conceptual difficulty with using a method which depends on an assumption of no variation in true treatment effect to estimate how the treatment effect varies with underlying risk. To illustrate further the danger of a simple analysis using some observed measure of “risk,” we consider briefly an approach using a graph of treatment effect against proportion of events in the treated group. The data from the sclerotherapy trials produce a slope of the fitted line of 0.51 (0.02 to 1.00), leading to the opposite conclusion from that obtained using the proportion of events in the control group. In other words, the danger is that any observed association between treatment effect and underlying risk is strongly dependent on the choice of measure of underlying risk, even to the extent of determining the direction of the...