Nonexperimental Versus Experimental Estimates of Earnings Impacts
- 1 September 2003
- journal article
- other
- Published by SAGE Publications in The Annals of the American Academy of Political and Social Science
- Vol. 589 (1) , 63-93
- https://doi.org/10.1177/0002716203254879
Abstract
To assess nonexperimental (NX) evaluation methods in the context of welfare, job training, and employment services programs, the authors reexamined the results of twelve case studies intended to replicate impact estimates from an experimental evaluation by using NX methods. They found that the NX methods sometimes came close to replicating experimentally derived results but often produced estimates that differed by policy-relevant margins, which the authors interpret as estimates of bias. Although the authors identified several study design factors associated with smaller discrepancies, no combination of factors would consistently eliminate discrepancies. Even with a large number of impact estimates, the positive and negative bias estimates did not always cancel each other out. Thus, it was difficult to identify an aggregation strategy that consistently removed bias while answering a focused question about earnings impacts of a program. They conclude that although the empirical evidence from this literature can be used in the context of training and welfare programs to improve NX research designs, it cannot on its own justify the use of such designs.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- Evaluating Social Policy by Experimental and Nonexperimental MethodsThe Scandinavian Journal of Economics, 2002
- Causal Effects in Nonexperimental Studies: Reevaluating the Evaluation of Training ProgramsJournal of the American Statistical Association, 1999
- Characterizing Selection Bias Using Experimental DataEconometrica, 1998
- Random versus nonrandom assignment in the evaluation of treatment for cocaine abusers.Journal of Consulting and Clinical Psychology, 1998
- Random versus nonrandom assignment in controlled experiments: Do you get the same answer?Journal of Consulting and Clinical Psychology, 1996
- Quasi-Experimental Estimates of the Effects of a Preschool InterventionEvaluation Review, 1995
- Effects of random versus nonrandom assignment in a comparison of inpatient and day hospital rehabilitation for male alcoholics.Journal of Consulting and Clinical Psychology, 1995
- The efficacy of psychological, educational, and behavioral treatment: Confirmation from meta-analysis.American Psychologist, 1993
- Choosing among Alternative Nonexperimental Methods for Estimating the Impact of Social Programs: The Case of Manpower TrainingJournal of the American Statistical Association, 1989
- The Adequacy of Comparison Group Designs for Evaluations of Employment-Related ProgramsThe Journal of Human Resources, 1987