Resolution-Based Complexity Control for Gaussian Mixture Models

1 February 2001

journal article
research article
Published by MIT Press in Neural Computation

Vol. 13 (2) , 453-475
https://doi.org/10.1162/089976601300014600

Abstract

In the domain of unsupervised learning, mixtures of gaussians have become a popular tool for statistical modeling. For this class of generative models, we present a complexity control scheme, which provides an effective means for avoiding the problem of overfitting usually encountered with unconstrained (mixtures of) gaussians in high dimensions. According to some prespecified level of resolution as implied by a fixed variance noise model, the scheme provides an automatic selection of the dimensionalities of some local signal subspaces by maximum likelihood estimation. Together with a resolution-based control scheme for adjusting the number of mixture components, we arrive at an incremental model refinement procedure within a common deterministic annealing framework, which enables an efficient exploration of the model space. The advantages of the resolution-based framework are illustrated by experimental results on synthetic and high-dimensional real-world data.

Keywords

This publication has 16 references indexed in Scilit:

Fast nonlinear dimension reduction
Published by Institute of Electrical and Electronics Engineers (IEEE) ,2002
Bayesian approaches to Gaussian mixture modeling
Published by Institute of Electrical and Electronics Engineers (IEEE) ,1998
Algorithms for Model-Based Gaussian Hierarchical Clustering
SIAM Journal on Scientific Computing, 1998
Unsupervised texture segmentation in a deterministic annealing framework
Published by Institute of Electrical and Electronics Engineers (IEEE) ,1998
Probabilistic visual learning for object representation
Published by Institute of Electrical and Electronics Engineers (IEEE) ,1997
Vector quantization by deterministic annealing
IEEE Transactions on Information Theory, 1992
Statistical mechanics and phase transitions in clustering
Physical Review Letters, 1990
An Analysis of the Elastic Net Approach to the Traveling Salesman Problem
Neural Computation, 1989
Nonparametric Maximum Likelihood Estimation by the Method of Sieves
The Annals of Statistics, 1982
Asymptotic Theory for Principal Component Analysis
The Annals of Mathematical Statistics, 1963