Graphical Models, Exponential Families, and Variational Inference
Foundations and Trends® in
Machine Learning
Volume 1 Issue 1–2
DOI: 10.1561/2200000001
Graphical Models, Exponential Families, and Variational Inference
Martin J. Wainwright
Department of Statistics, and Department of Electrical Engineering and
Computer Science, University of California, Berkeley 94720, USA, wainwrig@stat.berkeley.edu
Michael I. Jordan
Department of Statistics, and Department of Electrical Engineering and
Computer Science, University of California, Berkeley 94720, USA, jordan@stat.berkeley.edu
SUGGESTED CITATION:
Martin J.
Wainwright
and
Michael I.
Jordan
(2008)
"Graphical Models, Exponential Families, and Variational Inference",
Foundations and Trends® in Machine Learning: Vol. 1: No 1–2, pp 1-305.
http:/dx.doi.org/10.1561/2200000001
Abstract
The formalism of probabilistic graphical models provides a unifying framework for capturing complex dependencies among random
variables, and building large-scale multivariate statistical models. Graphical models have become a focus of research in many
statistical, computational and mathematical fields, including bioinformatics, communication theory, statistical physics, combinatorial
optimization, signal and image processing, information retrieval and statistical machine learning. Many problems that arise
in specific instances -- including the key problems of computing marginals and modes of probability distributions -- are best
studied in the general setting. Working with exponential family representations, and exploiting the conjugate duality between
the cumulant function and the entropy for exponential families, we develop general variational representations of the problems
of computing likelihoods, marginal probabilities and most probable configurations. We describe how a wide variety of algorithms --
among them sum-product, cluster variational methods, expectation-propagation, mean field methods, max-product and linear programming
relaxation, as well as conic programming relaxations -- can all be understood in terms of exact or approximate forms of these
variational representations. The variational approach provides a complementary alternative to Markov chain Monte Carlo as
a general source of approximation methods for inference in large-scale statistical models.