Finite-size effects in learning and generalization in linear perceptrons
- 7 December 1994
- journal article
- Published by IOP Publishing in Journal of Physics A: General Physics
- Vol. 27 (23) , 7771-7784
- https://doi.org/10.1088/0305-4470/27/23/020
Abstract
Most properties of learning and generalization in linear perceptrons can be derived from the average response function G. We present a method for calculating G using only simple matrix identities and partial differential equations. Using this method, we first rederive the known result for G in the thermodynamic limit of perceptrons of infinite size N, which has previously been calculated using replica and diagrammatic methods. We also show explicitly that the response function is self-averaging in the thermodynamic limit. Extensions of our method to more general learning scenarios with anisotropic teacher-space priors, input distributions, and weight-decay terms are discussed. Finally, finite-size effects are considered by calculating the O(1/N) correction to G. We verify the result by computer simulations and discuss the consequences for generalization and learning dynamics in linear perceptrons of finite size.Keywords
This publication has 6 references indexed in Scilit:
- Stochastic linear learning: Exact test and training error averagesNeural Networks, 1993
- Generalization in a linear perceptron in the presence of noiseJournal of Physics A: General Physics, 1992
- Learning with noise in a linear perceptronJournal of Physics A: General Physics, 1992
- Finite-size effects and bounds for perceptron modelsJournal of Physics A: General Physics, 1991
- Phase transitions in simple learningJournal of Physics A: General Physics, 1989
- Learning in Neural Networks: Solvable DynamicsEurophysics Letters, 1989