On-line versus off-line learning in the linear perceptron: A comparative study

The spherical perceptron with N inputs and a linear output does not present optimal generalization if trained by minimization of the standard quadratic cost function E=1/2 ${\mathcal{J}}_{\mathrm{\mu }=1\mathrm{}}^{\mathrm{\alpha }\mathit{N}}$ (${\mathit{b}}_{\mathrm{\mu }}$-${\mathit{h}}_{\mathrm{\mu }}$ ${)}^2$, where ${\mathit{b}}_{\mathrm{\mu }}$ and ${\mathit{h}}_{\mathrm{\mu }}$ are the outputs from the rule (teacher) and hypothesis (student) networks for the example μ and there are αN examples. We derive an optimal algorithm for on-line learning of examples which outperforms the iterative (off-line) standard algorithm for α up to 0.71. The on-line optimized algorithm suggests a class of cost functions for off-line learning, which we then proceed to study using the replica method. The optimized cost function within that class has the suggestive form E=αN[Γ(1/αN) ${\mathcal{J}}_{\mathrm{\mu }=1\mathrm{}}^{\mathrm{\alpha }\mathit{N}}$ [-lnP(${\mathit{b}}_{\mathrm{\mu }}$‖${\mathit{h}}_{\mathrm{\mu }}$)]-Γ lnZ], where Z is a normalization constant, P(${\mathit{b}}_{\mathrm{\mu }}$‖${\mathit{h}}_{\mathrm{\mu }}$) is the conditional probability of the output data ${\mathit{b}}_{\mathrm{\mu }}$ given the hypothesis output ${\mathit{h}}_{\mathrm{\mu }}$, and Γ is a learning parameter analogous to a temperature which decreases in a well defined manner along the learning process.