Can the Randomized Controlled Trial Literature Generalize to Nonrandomized Patients?

Publisher Website
Google Scholar
Abstract
To determine the extent to which published randomized controlled trials (RCTs) of psychotherapy can be generalized to a sample of outpatients, the authors matched information obtained from charts of patients who had been screened out of RCTs to inclusion and exclusion criteria from published RCT studies. Most of the patients in the sample who had primary diagnoses represented in the RCT literature were judged eligible for at least 1 RCT. However, many patients in the sample with substance use disorders or social anxiety disorder were not eligible for at least 2 RCTs. Common reasons that patients did not match with at least 2 published RCTs for psychotherapy included (a) patients were in partial remission, (b) patients failed to meet minimum severity or duration criteria, (c) patients were being treated with antidepressant medication, and (d) the disorder being studied was not primary (mostly for social anxiety patients). The implications of these findings for future research and clinical practice are discussed. Several authors have raised the concern that randomized con- trolled trials (RCTs) designed to evaluate the efficacy of psycho- therapy have typically had stringent exclusion criteria, potentially limiting the generalizability of the findings from such studies. For example, Westen and Morrison (2001) found that exclusion rates for RCTs for three common disorders were 68% for depression, 64% for panic, and 65% for generalized anxiety disorder (GAD). They expressed the concern that exclusion criteria for these dis- orders "often eliminated more troubled and difficult-to-treat pa- tients" (Westen & Morrison, 2001, p. 880), a concern shared by others (Goldfried & Wolfe, 1998; Persons & Silberschatz, 1998; Seligman, 1995). However, specific information about the simi- larities and differences among patients who participate in RCTs and those seen in clinical practice cannot be determined by exam- ining screen-out rates of individual studies. The majority of pub- lished studies do not disclose the percentages of potential partic- ipants who were eliminated for various reasons. Therefore, those reasons remain the source of speculation. In a recent study (Stirman, DeRubeis, Crits-Christoph, & Brody, 2003), the characteristics of patients seen in private practice were compared with the inclusion and exclusion criteria for RCTs for psychotherapies in an attempt to determine what types of patients might be less likely to be addressed in the RCT literature. Stirman and colleagues found that nearly half of the patients in their sample would not be included in research because they had diagnoses that were not studied in the literature, most commonly adjustment disorder. However, the majority of patients in their sample who had diagnoses that were studied in the psychotherapy outcome literature would have been eligible for at least two published RCTs. Among those who were judged to be ineligible, insufficient severity was listed as the most common reason for ineligibility. Despite the 42% rate of comorbidity in their sample, only 3% of Stirman et al.'s sample of patients who had diagnoses represented in the literature were judged to be ineligible on the basis of comorbidity. Less than 1% of the sample was judged ineligible because their case was deemed "too severe." These findings led Stirman et al. (2003) to conclude that the RCT literature might actually overrepresent those with more se- vere forms of many of the diagnoses studied. Such findings have implications for both researchers and clinicians. On the basis of these findings, researchers who wish to increase the applicability of their research to clinical practice might broaden the scope of their studies to include depression-spectrum disorders and less severe cases. Stirman et al. also suggested that RCT studies can be used to inform the treatment of individual cases, even for patients with more severe or comorbid diagnoses. By comparing individual patient characteristics with the inclusion criteria of RCT studies, clinicians can maximize the relevance of the research to a partic- ular patient. However, Stirman and colleagues (2003) noted that their find- ings should be interpreted with caution, as the diagnostic informa- tion used in the study was obtained through unstructured inter- views, and the reliability of the diagnoses was therefore questionable. Moreover, an extremely low rate of Axis II comor- bidity was observed, and their sample was relatively highly edu- cated and was greater than 80% Caucasian. Thus, further study, using a sample of patients with reliable Axis I and Axis II diag- noses and different demographic characteristics, is necessary be- fore the findings of Stirman et al. can be accepted with confidence. In the present study, we replicated the procedures used in the previous study, focusing on patients who were screened out of psychotherapy outcome studies at the University of Pennsylvania.