Weight smoothing to improve network generalization
- 1 January 1994
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks
- Vol. 5 (5) , 752-763
- https://doi.org/10.1109/72.317727
Abstract
A weight smoothing algorithm is proposed in this paper to improve a neural network's generalization capability. The algorithm can be used when the data patterns to be classified are presented on an n-dimensional grid (n⩾1) and there exists some correlations among neighboring data points within a pattern. For a fully-interconnected feedforward net, no such correlation information is embedded into the architecture. Consequently, the correlations can only be extracted through sufficient amount of network training. With the proposed algorithm, a smoothing constraint is incorporated into the objective function of backpropagation to reflect the neighborhood correlations and to seek those solutions that have smooth connection weights. Experiments were performed on problems of waveform classification, multifont alphanumeric character recognition, and handwritten numeral recognition. The results indicate that (1) networks trained with the algorithm do have smooth connection weights, and (2) they generalize betterKeywords
This publication has 16 references indexed in Scilit:
- Neocognitron: A hierarchical neural network capable of visual pattern recognitionPublished by Elsevier ,2003
- Handwritten zip code recognition with multilayer networksPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2002
- A new distance measure for binary imagesPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2002
- Translation, rotation, and scale invariant pattern recognition by high-order neural networks and moment classifiersIEEE Transactions on Neural Networks, 1992
- Encoding a priori information in feedforward networksNeural Networks, 1991
- A study of methods of choosing the smoothing parameter in image restoration by regularizationPublished by Institute of Electrical and Electronics Engineers (IEEE) ,1991
- A neural network approach to character recognitionNeural Networks, 1989
- Backpropagation Applied to Handwritten Zip Code RecognitionNeural Computation, 1989
- Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of ImagesPublished by Institute of Electrical and Electronics Engineers (IEEE) ,1984
- Optimization by Simulated AnnealingScience, 1983