Asymptotic statistical theory of overtraining and cross-validation
- 1 September 1997
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks
- Vol. 8 (5) , 985-996
- https://doi.org/10.1109/72.623200
Abstract
A statistical theory for overtraining is proposed. The analysis treats general realizable stochastic neural networks, trained with Kullback-Leibler divergence in the asymptotic case of a large number of training examples. It is shown that the asymptotic gain in the generalization error is small if we perform early stopping, even if we have access to the optimal stopping time. Based on the cross-validation stopping we consider the ratio the examples should be divided into training and cross-validation sets in order to obtain the optimum performance. Although cross-validated early stopping is useless in the asymptotic region, it surely decreases the generalization error in the nonasymptotic region. Our large scale simulations done on a CM5 are in good agreement with our analytical findings.Keywords
This publication has 15 references indexed in Scilit:
- A Bound on the Error of Cross Validation Using the Approximation and Estimation Rates, with Consequences for the Training-Test SplitNeural Computation, 1997
- On-line learning in soft committee machinesPhysical Review E, 1995
- Test Error Fluctuations in Finite Linear PerceptronsNeural Computation, 1995
- Finite-size effects and optimal test set size in linear perceptronsJournal of Physics A: General Physics, 1995
- The Nature of Statistical Learning TheoryPublished by Springer Nature ,1995
- Statistical Theory of Learning Curves under Entropic Loss CriterionNeural Computation, 1993
- An Introduction to the BootstrapPublished by Springer Nature ,1993
- Generalization in a linear perceptron in the presence of noiseJournal of Physics A: General Physics, 1992
- Learning processes in neural networksPhysical Review A, 1991
- A new look at the statistical model identificationIEEE Transactions on Automatic Control, 1974