Learning unrealizable tasks from minimum entropy queries
- 7 November 1995
- journal article
- Published by IOP Publishing in Journal of Physics A: General Physics
- Vol. 28 (21) , 6125-6142
- https://doi.org/10.1088/0305-4470/28/21/016
Abstract
In supervised learning, learning from queries rather than from random examples can improve generalization performance significantly. We study the performance of query learning for unrealizable tasks, where the student cannot learn from the perfectly. As a simple model scenario of this kind, we consider a linear perceptron student learning a general nonlinear perceptron teacher. Two kinds of queries for maximum information gain, i.e. minimum entropy, are investigated: minimum student space entropy (MSSE) queries, which are appropriate if the teacher space is unknown, and minimum teacher space entropy (MTSE) queries, which can be used if the teacher space is assumed to be known, but a student of a simpler form has deliberately been chosen. We find that for MSSE queries, the structure of the student space determines the efficacy of query learning. MTSE queries, on the other hand, which we investigate for the extreme case of a binary perceptron teacher, lead to a higher generalization error than random examples, due to a lack of feedback about the progress of the student in the way queries are selected.Keywords
This publication has 16 references indexed in Scilit:
- Learning in linear neural networks: a surveyIEEE Transactions on Neural Networks, 1995
- Finite-size effects and optimal test set size in linear perceptronsJournal of Physics A: General Physics, 1995
- Statistical mechanics of hypothesis evaluationJournal of Physics A: General Physics, 1994
- Query construction, entropy, and generalization in neural-network modelsPhysical Review E, 1994
- Learning and generalization in a linear perceptron stochastically trained with noisy dataJournal of Physics A: General Physics, 1993
- The statistical mechanics of learning a ruleReviews of Modern Physics, 1993
- Generalization ability of perceptrons with continuous outputsPhysical Review E, 1993
- Statistical mechanics of learning from examplesPhysical Review A, 1992
- Generalization in a linear perceptron in the presence of noiseJournal of Physics A: General Physics, 1992
- A statistical approach to learning and generalization in layered neural networksProceedings of the IEEE, 1990