Storage capacity and learning algorithms for two-layer neural networks

A two-layer feedforward network of McCulloch-Pitts neurons with N inputs and K hidden units is analyzed for N→∞ and K finite with respect to its ability to implement p=αN random input-output relations. Special emphasis is put on the case where all hidden units are coupled to the output with the same strength (committee machine) and the receptive fields of the hidden units either enclose all input units (fully connected) or are nonoverlapping (tree structure). The storage capacity is determined generalizing Gardner’s treatment [J. Phys. A 21, 257 (1988); Europhys. Lett. 4, 481 (1987)] of the single-layer perceptron. For the treelike architecture, a replica-symmetric calculation yields ${\mathrm{\alpha }}_{\mathit{c}}$∝ √K for a large number K of hidden units. This result violates an upper bound derived by Mitchison and Durbin [Biol. Cybern. 60, 345 (1989)]. One-step replica-symmetry breaking gives lower values of ${\mathrm{\alpha }}_{\mathit{c}}$.