On the Geometry of Feedforward Neural Network Error Surfaces
- 1 November 1993
- journal article
- Published by MIT Press in Neural Computation
- Vol. 5 (6) , 910-927
- https://doi.org/10.1162/neco.1993.5.6.910
Abstract
Many feedforward neural network architectures have the propertythat their overall input-output function is unchanged by certainweight permutations and sign flips. In this paper, the geometricstructure of these equioutput weight space transformationsis explored for the case of multilayer perceptron networks withtanh activation functions (similar results hold for manyother types of neural networks). It is shown that thesetransformations form an algebraic group isomorphic to a directproduct of Weyl groups. Results concerning the root spaces of theLie algebras associated with these Weyl groups are then used toderive sets of simple equations for minimal sufficient search setsin weight space. These sets, which take the geometric forms of awedge and a cone, occupy only a minute fraction of the volume ofweight space. A separate analysis shows that large numbers ofcopies of a network performance function optimum weight vector arecreated by the action of the equioutput transformation group andthat these copies all lie on the same sphere. Some implications ofthese results for learning are discussed.Keywords
This publication has 10 references indexed in Scilit:
- Uniqueness of the weights for minimal feedforward nets with a given input-output mapNeural Networks, 1992
- Self-organizing neural network that discovers surfaces in random-dot stereogramsNature, 1992
- Improving the Generalization Properties of Radial Basis Function Neural NetworksNeural Computation, 1991
- ARTMAP: Supervised real-time learning and classification of nonstationary data by a self-organizing neural networkNeural Networks, 1991
- Layered Neural Networks with Gaussian Hidden Units as Universal ApproximationsNeural Computation, 1990
- Regularization Algorithms for Learning That Are Equivalent to Multilayer NetworksScience, 1990
- Curvature-Driven Smoothing in Backpropagation Neural NetworksPublished by Springer Nature ,1990
- Fast Learning in Networks of Locally-Tuned Processing UnitsNeural Computation, 1989
- Self-organization in a perceptual networkComputer, 1988
- A neural model for category learningBiological Cybernetics, 1982