A Unifying Review of Linear Gaussian Models
- 1 February 1999
- journal article
- review article
- Published by MIT Press in Neural Computation
- Vol. 11 (2) , 305-345
- https://doi.org/10.1162/089976699300016674
Abstract
Factor analysis, principal component analysis, mixtures of gaussian clusters, vector quantization, Kalman filter models, and hidden Markov models can all be unified as variations of unsupervised learning under a single basic generative model. This is achieved by collecting together disparate observations and derivations made by many previous authors and introducing a new way of linking discrete and continuous state models using a simple nonlinearity. Through the use of other nonlinearities, we show how independent component analysis is also a variation of the same basic generative model. We show that factor analysis and mixtures of gaussians can be implemented in autoencoder neural networks and learned using squared error plus the same regularization term. We introduce a new model for static data, known as sensible principal component analysis, as well as a novel concept of spatially adaptive observation noise. We also review some of the literature involving global and local mixtures of the basic models and provide pseudocode for inference and learning for all the basic models.Keywords
This publication has 32 references indexed in Scilit:
- Blind Source Separation and Deconvolution: The Dynamic Component Analysis AlgorithmNeural Computation, 1998
- GTM: The Generative Topographic MappingNeural Computation, 1998
- An Information-Maximization Approach to Blind Separation and Blind DeconvolutionNeural Computation, 1995
- Independent component analysis, A new concept?Signal Processing, 1994
- ML estimation of a stochastic linear system with the EM algorithm and its application to speech recognitionIEEE Transactions on Speech and Audio Processing, 1993
- Neural networks and principal component analysis: Learning from examples without local minimaNeural Networks, 1989
- A Maximization Technique Occurring in the Statistical Analysis of Probabilistic Functions of Markov ChainsThe Annals of Mathematical Statistics, 1970
- An inequality with applications to statistical estimation for probabilistic functions of Markov processes and to a model for ecologyBulletin of the American Mathematical Society, 1967
- Statistical Inference for Probabilistic Functions of Finite State Markov ChainsThe Annals of Mathematical Statistics, 1966
- A Rapidly Convergent Descent Method for MinimizationThe Computer Journal, 1963