Information-theoretic upper and lower bounds for statistical estimation
- 3 April 2006
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Information Theory
- Vol. 52 (4) , 1307-1321
- https://doi.org/10.1109/tit.2005.864439
Abstract
In this paper, we establish upper and lower bounds for some statistical estimation problems through concise information-theoretic arguments. Our upper bound analysis is based on a simple yet general inequality which we call the information exponential inequality. We show that this inequality naturally leads to a general randomized estimation method, for which performance upper bounds can be obtained. The lower bounds, applicable for all statistical estimators, are obtained by original applications of some well known information-theoretic inequalities, and approximately match the obtained upper bounds for various important problems. Moreover, our framework can be regarded as a natural generalization of the standard minimax framework, in that we allow the performance of the estimator to vary for different possible underlying distributions according to a predefined priorKeywords
This publication has 9 references indexed in Scilit:
- PAC-Bayesian Generalisation Error Bounds for Gaussian Process ClassificationPublished by Test accounts ,2003
- Information-theoretic determination of minimax rates of convergenceThe Annals of Statistics, 1999
- PAC-Bayesian model averagingPublished by Association for Computing Machinery (ACM) ,1999
- Weak Convergence and Empirical ProcessesPublished by Springer Nature ,1996
- Generalizing the Fano inequalityIEEE Transactions on Information Theory, 1994
- Rates of convergence for minimum contrast estimatorsProbability Theory and Related Fields, 1993
- Minimum complexity density estimationIEEE Transactions on Information Theory, 1991
- Information bounds of the Fano-Kullback typeIEEE Transactions on Information Theory, 1976
- Probability Inequalities for Sums of Bounded Random VariablesJournal of the American Statistical Association, 1963