Gradient descent learning algorithm overview: a general dynamical systems perspective
- 1 January 1995
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks
- Vol. 6 (1) , 182-195
- https://doi.org/10.1109/72.363438
Abstract
Gives a unified treatment of gradient descent learning algorithms for neural networks using a general framework of dynamical systems. This general approach organizes and simplifies all the known algorithms and results which have been originally derived for different problems (fixed point/trajectory learning), for different models (discrete/continuous), for different architectures (forward/recurrent), and using different techniques (backpropagation, variational calculus, adjoint methods, etc.). The general approach can also be applied to derive new algorithms. The author then briefly examines some of the complexity issues and limitations intrinsic to gradient descent learning. Throughout the paper, the author focuses on the problem of trajectory learning.Keywords
This publication has 25 references indexed in Scilit:
- How delays affect neural dynamics and learningIEEE Transactions on Neural Networks, 1994
- Neural Networks and the Bias/Variance DilemmaNeural Computation, 1992
- A general weight matrix formulation using optimal controlIEEE Transactions on Neural Networks, 1991
- Computing with Arrays of Coupled Oscillators: An Application to Preattentive Texture DiscriminationNeural Computation, 1990
- Faster Learning for Dynamic Recurrent BackpropagationNeural Computation, 1990
- A Learning Algorithm for Continually Running Fully Recurrent Neural NetworksNeural Computation, 1989
- Learning State Space Trajectories in Recurrent Neural NetworksNeural Computation, 1989
- Convergence results in an associative memory modelNeural Networks, 1988
- A model of neuronal bursting using three coupled first order differential equationsProceedings of the Royal Society of London. B. Biological Sciences, 1984
- A Steepest-Ascent Method for Solving Optimum Programming ProblemsJournal of Applied Mechanics, 1962