Two-layer perceptrons at saturation

Publisher Website
Google Scholar
Abstract
We study multilayer networks which implement a fixed Boolean function from the hidden layer to the output, and have a fully connected architecture from the input to the hidden layer. We analyze the organization of the first layer of weights and the capacity of the network by a statistical mechanical approach. The mean-field equations that govern the behavior of a network at saturation are derived assuming a replica symmetric solution. The theory is applied for a detailed analysis of the learning ability of an AND machine. We calculate the maximal capacity, and the overlap between the subnetworks. Attention is paid to the organization of internal representations at saturation. The results are compared with detailed numerical simulations, and with bounds on the capacity. Good agreement is found. The behavior of the system depends on the ratio between the number of patterns with a (+) and (-) output. We compare the fully connected architecture with networks that have nonoverlapping receptive fields from the input to the hidden layer. The mean-field equations for this architecture are also presented.