Diagrammatic Derivation of Gradient Algorithms for Neural Networks
- 1 January 1996
- journal article
- Published by MIT Press in Neural Computation
- Vol. 8 (1) , 182-201
- https://doi.org/10.1162/neco.1996.8.1.182
Abstract
Deriving gradient algorithms for time-dependent neural network structures typically requires numerous chain rule expansions, diligent bookkeeping, and careful manipulation of terms. In this paper, we show how to derive such algorithms via a set of simple block diagram manipulation rules. The approach provides a common framework to derive popular algorithms including backpropagation and backpropagation-through-time without a single chain rule expansion. Additional examples are provided for a variety of complicated architectures to illustrate both the generality and the simplicity of the approach.Keywords
This publication has 11 references indexed in Scilit:
- Neural networks for feedback feedforward nonlinear control systemsIEEE Transactions on Neural Networks, 1994
- Relating Real-Time Backpropagation and Backpropagation-Through-Time: An Application of Flow Graph InterreciprocityNeural Computation, 1994
- Neural Networks and Nonlinear Adaptive Filtering: Unifying Concepts and New AlgorithmsNeural Computation, 1993
- FIR and IIR Synapses, a New Neural Network Architecture for Time Series ModelingNeural Computation, 1991
- Learning of neural networks using their adjoint systemsSystems and Computers in Japan, 1991
- Identification and control of dynamical systems using neural networksIEEE Transactions on Neural Networks, 1990
- Backpropagation Applied to Handwritten Zip Code RecognitionNeural Computation, 1989
- A Learning Algorithm for Continually Running Fully Recurrent Neural NetworksNeural Computation, 1989
- Multilayer feedforward networks are universal approximatorsNeural Networks, 1989
- Inter-reciprocity applied to electrical networksApplied Scientific Research, Section B, 1957