Pruning recurrent neural networks for improved generalization performance
- 1 September 1994
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks
- Vol. 5 (5) , 848-851
- https://doi.org/10.1109/72.317740
Abstract
Determining the architecture of a neural network is an important issue for any learning task. For recurrent neural networks no general methods exist that permit the estimation of the number of layers of hidden neurons, the size of layers or the number of weights. We present a simple pruning heuristic that significantly improves the generalization performance of trained recurrent networks. We illustrate this heuristic by training a fully recurrent neural network on positive and negative strings of a regular grammar. We also show that rules extracted from networks trained with this pruning heuristic are more consistent with the rules to be learned. This performance improvement is obtained by pruning and retraining the networks. Simulations are shown for training and pruning a recurrent neural net on strings generated by two regular grammars, a randomly-generated 10-state grammar and an 8-state, triple-parity grammar. Further simulations indicate that this pruning method can have generalization performance superior to that obtained by training with weight decay.<>Keywords
This publication has 12 references indexed in Scilit:
- Heuristics for the extraction of rules from discrete-time recurrent neural networksPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2003
- Inserting rules into recurrent neural networksPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2003
- Combining symbolic and neural learningMachine Learning, 1994
- Learning Finite State Machines With Self-Clustering Recurrent NetworksNeural Computation, 1993
- Extraction, Insertion and Refinement of Symbolic Rules in Dynamically Driven Recurrent Neural NetworksConnection Science, 1993
- Induction of Finite-State Languages Using Second-Order Recurrent NetworksNeural Computation, 1992
- Learning and Extracting Finite State Automata with Second-Order Recurrent Neural NetworksNeural Computation, 1992
- The induction of dynamical recognizersMachine Learning, 1991
- An unified approach for integrating explicit knowledge and learning by example in recurrent networksPublished by Institute of Electrical and Electronics Engineers (IEEE) ,1991
- A Learning Algorithm for Continually Running Fully Recurrent Neural NetworksNeural Computation, 1989