High-order and multilayer perceptron initialization
- 1 March 1997
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks
- Vol. 8 (2) , 349-359
- https://doi.org/10.1109/72.557673
Abstract
Proper initialization is one of the most important prerequisites for fast convergence of feedforward neural networks like high-order and multilayer perceptrons. This publication aims at determining the optimal variance (or range) for the initial weights and biases, which is the principal parameter of random initialization methods for both types of neural networks. An overview of random weight initialization methods for multilayer perceptrons is presented. These methods are extensively tested using eight real-world benchmark data sets and a broad range of initial weight variances by means of more than 30000 simulations, in the aim to find the best weight initialization method for multilayer perceptrons. For high-order networks, a large number of experiments (more than 200000 simulations) was performed, using three weight distributions, three activation functions, several network orders, and the same eight data sets. The results of these experiments are compared to weight initialization techniques for multilayer perceptrons, which leads to the proposal of a suitable initialization method for high-order perceptrons. The conclusions on the initialization methods for both types of networks are justified by sufficiently small confidence intervals of the mean convergence times.Keywords
This publication has 14 references indexed in Scilit:
- The Interchangeability of Learning Rate and Gain in Backpropagation Neural NetworksNeural Computation, 1996
- Adaptive multilayer optical neural network with optical thresholdingOptical Engineering, 1995
- Modular Object-Oriented Neural Network Simulators and Topology GeneralizationsPublished by Springer Nature ,1994
- Initializing back propagation networks with prototypesNeural Networks, 1993
- Do Backpropagation Trained Neural Networks have Normal Weight Distributions?Published by Springer Nature ,1993
- An analysis of premature saturation in back propagation learningNeural Networks, 1993
- Statistically controlled activation weight initialization (SCAWI)IEEE Transactions on Neural Networks, 1992
- Avoiding false local minima by proper initialization of connectionsIEEE Transactions on Neural Networks, 1992
- Weight value initialization for improving training speed in the backpropagation networkPublished by Institute of Electrical and Electronics Engineers (IEEE) ,1991
- Machine Learning Using a Higher Order Correlation NetworkPhysica D: Nonlinear Phenomena, 1986