Computing second derivatives in feed-forward networks: a review
- 1 May 1994
- journal article
- review article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks
- Vol. 5 (3) , 480-488
- https://doi.org/10.1109/72.286919
Abstract
The calculation of second derivatives is required by recent training and analysis techniques of connectionist networks, such as the elimination of superfluous weights, and the estimation of confidence intervals both for weights and network outputs. We review and develop exact and approximate algorithms for calculating second derivatives. For networks with |w| weights, simply writing the full matrix of second derivatives requires O(|w|(2)) operations. For networks of radial basis units or sigmoid units, exact calculation of the necessary intermediate terms requires of the order of 2h+2 backward/forward-propagation passes where h is the number of hidden units in the network. We also review and compare three approximations (ignoring some components of the second derivative, numerical differentiation, and scoring). The algorithms apply to arbitrary activation functions, networks, and error functions.Keywords
This publication has 8 references indexed in Scilit:
- Fast Exact Multiplication by the HessianNeural Computation, 1994
- A scaled conjugate gradient algorithm for fast supervised learningNeural Networks, 1993
- Exact Calculation of the Hessian Matrix for the Multilayer PerceptronNeural Computation, 1992
- A Practical Bayesian Framework for Backpropagation NetworksNeural Computation, 1992
- A new error criterion for posterior probability estimation with neural netsPublished by Institute of Electrical and Electronics Engineers (IEEE) ,1990
- PREDICTING THE FUTURE: A CONNECTIONIST APPROACHInternational Journal of Neural Systems, 1990
- Generalized Linear ModelsPublished by Springer Nature ,1989
- Increased rates of convergence through learning rate adaptationNeural Networks, 1988