On overfitting, generalization, and randomly expanded training sets

1 September 2000

journal article
Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks

Vol. 11 (5) , 1050-1057
https://doi.org/10.1109/72.870038

Abstract

An algorithmic procedure is developed for the random expansion of a given training set to combat overfitting and improve the generalization ability of backpropagation trained multilayer perceptrons (MLPs). The training set is K-means clustered and locally most entropic colored Gaussian joint input-output probability density function estimates are formed per cluster. The number of clusters is chosen such that the resulting overall colored Gaussian mixture exhibits minimum differential entropy upon global cross-validated shaping. Numerical studies on real data and synthetic data examples drawn from the literature illustrate and support these theoretical developments.

Keywords

This publication has 26 references indexed in Scilit:

Pruning algorithms-a survey
IEEE Transactions on Neural Networks, 1993
Enhanced training algorithms, and integrated training/architecture selection for multilayer perceptron networks
IEEE Transactions on Neural Networks, 1992
Multivariate Adaptive Regression Splines
The Annals of Statistics, 1991
An Asymptotically Efficient Solution to the Bandwidth Problem of Kernel Density Estimation
The Annals of Statistics, 1985
Vector quantization in speech coding
Proceedings of the IEEE, 1985
Consistent Cross-Validated Density Estimation
The Annals of Statistics, 1983
Modeling by shortest data description
Automatica, 1978
Estimating the Dimension of a Model
The Annals of Statistics, 1978
On the Choice of Smoothing Parameters for Parzen Estimators of Probability Density Functions
IEEE Transactions on Computers, 1976
Statistical predictor identification
Annals of the Institute of Statistical Mathematics, 1970