Statistical mechanics of ensemble learning
- 1 January 1997
- journal article
- research article
- Published by American Physical Society (APS) in Physical Review E
- Vol. 55 (1) , 811-825
- https://doi.org/10.1103/physreve.55.811
Abstract
Within the context of learning a rule from examples, we study the general characteristics of learning with ensembles. The generalization performance achieved by a simple model ensemble of linear students is calculated exactly in the thermodynamic limit of a large number of input components and shows a surprisingly rich behavior. Our main findings are the following. For learning in large ensembles, it is advantageous to use underregularized students, which actually overfit the training data. Globally optimal generalization performance can be obtained by choosing the training set sizes of the students optimally. For smaller ensembles, optimization of the ensemble weights can yield significant improvements in ensemble generalization performance, in particular if the individual students are subject to noise in the training process. Choosing students with a wide range of regularization parameters makes this improvement robust against changes in the unknown level of corruption of the training data.This publication has 20 references indexed in Scilit:
- Boosting a Weak Learning Algorithm by MajorityInformation and Computation, 1995
- Improving model accuracy using optimal linear combinations of trained neural networksIEEE Transactions on Neural Networks, 1995
- Statistical mechanics of hypothesis evaluationJournal of Physics A: General Physics, 1994
- The statistical mechanics of learning a ruleReviews of Modern Physics, 1993
- A Practical Bayesian Framework for Backpropagation NetworksNeural Computation, 1992
- Neural Networks and the Bias/Variance DilemmaNeural Computation, 1992
- Stacked generalizationNeural Networks, 1992
- Adaptive Mixtures of Local ExpertsNeural Computation, 1991
- Neural network ensemblesPublished by Institute of Electrical and Electronics Engineers (IEEE) ,1990
- Invited review combining forecasts—twenty years laterJournal of Forecasting, 1989