An efficient constrained training algorithm for feedforward networks
- 1 January 1995
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks
- Vol. 6 (6) , 1420-1434
- https://doi.org/10.1109/72.471365
Abstract
A novel algorithm is presented which supplements the training phase in feedforward networks with various forms of information about desired learning properties. This information is represented by conditions which must be satisfied in addition to the demand for minimization of the usual mean square error cost function. The purpose of these conditions is to improve convergence, learning speed, and generalization properties through prompt activation of the hidden units, optimal alignment of successive weight vector offsets, elimination of excessive hidden nodes, and regulation of the magnitude of search steps in the weight space. The algorithm is applied to several small- and large-scale binary benchmark training tasks, to test its convergence ability and learning speed, as well as to a large-scale OCR problem, to test its generalization capability. Its performance in terms of percentage of local minima, learning speed, and generalization ability is evaluated and found superior to the performance of the backpropagation algorithm and variants thereof taking especially into account the statistical significance of the results.Keywords
This publication has 32 references indexed in Scilit:
- Training feed-forward networks with the extended Kalman algorithmPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2003
- Optimal filtering algorithms for fast learning in feedforward neural networksNeural Networks, 1992
- Neural Network Approaches Versus Statistical Methods In Classification Of Multisource Remote Sensing DataIEEE Transactions on Geoscience and Remote Sensing, 1990
- Backpropagation through time: what it does and how to do itProceedings of the IEEE, 1990
- Handwritten digit recognition: applications of neural network chips and automatic learningIEEE Communications Magazine, 1989
- On the complexity of loading shallow neural networksJournal of Complexity, 1988
- Increased rates of convergence through learning rate adaptationNeural Networks, 1988
- Analysis of hidden units in a layered network trained to classify sonar targetsNeural Networks, 1988
- Neutral networks for computing?AIP Conference Proceedings, 1986
- Connectionist Models and Their Applications: IntroductionCognitive Science, 1985