On-line learning with minimal degradation in feedforward networks
- 1 May 1995
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks
- Vol. 6 (3) , 657-668
- https://doi.org/10.1109/72.377971
Abstract
Dealing with non-stationary processes requires quick adaptation while at the same time avoiding catastrophic forgetting. A neural learning technique that satisfies these requirements, without sacrifying the benefits of distributed representations, is presented. It relies on a formalization of the problem as the minimization of the error over the previously learned input-output (i-o) patterns, subject to the constraint of perfect encoding of the new pattern. Then this constrained optimization problem is transformed into an unconstrained one with hidden-unit activations as variables. This new formulation naturally leads to an algorithm for solving the problem, which we call Learning with Minimal Degradation (LMD). Some experimental comparisons of the performance of LMD with back-propagation are provided which, besides showing the advantages of using LMD, reveal the dependence of forgetting on the learning rate in back-propagation. We also explain why overtraining affects forgetting and fault-tolerance, which are seen as related problems.Peer RevieweKeywords
This publication has 8 references indexed in Scilit:
- A Resource-Allocating Network for Function InterpolationNeural Computation, 1991
- An adaptively trained neural networkIEEE Transactions on Neural Networks, 1991
- Networks for approximation and learningProceedings of the IEEE, 1990
- A simple procedure for pruning back-propagation trained neural networksIEEE Transactions on Neural Networks, 1990
- Neural Network Design and the Complexity of LearningPublished by MIT Press ,1990
- Connectionist models of recognition memory: Constraints imposed by learning and forgetting functions.Psychological Review, 1990
- Fast Learning in Networks of Locally-Tuned Processing UnitsNeural Computation, 1989
- Neural nets for adaptive filtering and adaptive pattern recognitionComputer, 1988