ENTROPY MINIMAX MULTIVARIATE STATISTICAL MODELING–I: THEORY

Publisher Website
Google Scholar
Abstract
The theory of entropy mimmax, an information theoretic approach to predictive modeling, is reviewed. Comparisons are given to other methodologies, including: Neyman-Pearson hypothesis testing, James-Stein “empirical Bayes” estimation, maximum likelihood estimation, least-squares fitting, linear regression and logistic regression. Examples are provided showing how maximum entropy expectation probabilities are computed and how minimum entropy partitions are determined. The importance of the a priori weight normalization, in establishing the coarse-grain of the minimum entropy partition, is discussed. The trial crossvalidation procedure for determining the normalization is described. Generalizations utilizing Zadeh's fuzzy entropies are provided for variables involving indistinguishability, partial or total. Specific cases are discussed of maximum likelihood estimation, illustrating ils “data range irregularity” which is avoided by methods such as entropy minimization that account for residuals over the full distribution range. Discriminant analysis and polynomial fitting are discussed as examples of areas of application of the principles of entropy minimax. In curve fitting, the order of the polynomial is determined by entropy minimization, and the coefficients are determined by entropy maximization. Entropy maximization operates as a goodness-of-fit criterion, while entropy minimization operates as a mathematical formulation of Ockham's razor, controlling the level of smoothness in fitting models to data.

This publication has 40 references indexed in Scilit: