Boosting for high-dimensional linear models
Top Cited Papers
Open Access
- 1 April 2006
- journal article
- Published by Institute of Mathematical Statistics in The Annals of Statistics
- Vol. 34 (2) , 559-583
- https://doi.org/10.1214/009053606000000092
Abstract
We prove that boosting with the squared error loss, L2Boosting, is consistent for very high-dimensional linear models, where the number of predictor variables is allowed to grow essentially as fast as O(exp(sample size)), assuming that the true underlying regression function is sparse in terms of the ℓ1-norm of the regression coefficients. In the language of signal processing, this means consistency for de-noising using a strongly overcomplete dictionary if the underlying signal is sparse in terms of the ℓ1-norm. We also propose here an AIC-based method for tuning, namely for choosing the number of boosting iterations. This makes L2Boosting computationally attractive since it is not required to run the algorithm multiple times for cross-validation as commonly used so far. We demonstrate L2Boosting for simulated data, in particular where the predictor dimension is large in comparison to sample size, and for a difficult tumor-classification problem with gene expression microarray data.Keywords
All Related Versions
This publication has 22 references indexed in Scilit:
- Boosting with early stopping: Convergence and consistencyThe Annals of Statistics, 2005
- Regularization and Variable Selection Via the Elastic NetJournal of the Royal Statistical Society Series B: Statistical Methodology, 2005
- Persistence in high-dimensional linear predictor selection and the virtue of overparametrizationBernoulli, 2004
- Complexity regularization via localized random penaltiesThe Annals of Statistics, 2004
- Least angle regressionThe Annals of Statistics, 2004
- Process consistency for AdaBoostThe Annals of Statistics, 2004
- Comparison of Discrimination Methods for the Classification of Tumors Using Gene Expression DataJournal of the American Statistical Association, 2002
- Adaptive Prediction and Estimation in Linear Regression with Infinitely Many ParametersThe Annals of Statistics, 2001
- Arcing classifier (with discussion and a rejoinder by the author)The Annals of Statistics, 1998
- Matching pursuits with time-frequency dictionariesIEEE Transactions on Signal Processing, 1993