The Exponential Family of Distributions

The Exponential Family of Distributions

Prof. Nicholas Zabaras School of Engineering University of Warwick

Coventry CV4 7AL United Kingdom

Email: nzabaras@ URL:

August 12, 2014

Bayesian Scientific Computing, Spring 2013 (N. Zabaras)

1

Contents

The Exponential Family

The Bernoulli Distribution (Introducing Logistic Sigmoid), The Poisson Distribution, The Multinomial Distribution (Introducing SoftMax)

The Beta Distribution, The Gamma Distribution, The Gaussian Distribution, The von Mises Distribution, The Multivariate Gaussian

Computing the Moments of a Distribution from the Exponential Family, Moment Parametrization, Sufficiency and Neymann Factorization, Sufficient Statistics and MLE Estimates, MLE and Kullback-Leibler Distance

Conjugate Priors, Posterior Predictive, Maximum Entropy and the Exponential Family

Generalized Linear Models, Canonical Response Function, Batch IRLS, Sequential Estimation - LMS

Chris Bishops' PRML book, Chapter 2. M. Jordan, An Introduction to Probabilistic Graphical Models, Chapter 8 (preprint)

Bayesian Scientific Computing, Spring 2013 (N. Zabaras)

2

Exponential Family

Large family of useful distributions with common properties

Bernoulli, beta, binomial, chi-square, Dirichlet, gamma, Gaussian, geometric, multinomial, Poisson, Weibull, . .

Not in the family: Uniform, Student's T, Cauchy, Laplace, mixture of Gaussians, . . .

Variable can be discrete or continuous (or vectors thereof)

We will focus on the conditional setting in which we have

a directed model XY with both X & Y observed, and

with p(Y|X) being an exponential family distribution parametrized using Generalized Linear Models (GLIM's).

Bayesian Scientific Computing, Spring 2013 (N. Zabaras)

3

Exponential Family

The exponential family of distributions over x, given parameters , is defined to be the set of distributions of the form

p( x |h) h( x)g(h) exp hTu( x) or

p( x |h) h( x) exp hTu( x) A(h) , where : A(h) log g(h)

x is scalar/vector, discrete/continuous. are the natural

parameters and u(x) is referred to as a sufficient statistic.

g() ensures that the distribution is normalized and satisfies

g(h) h(x) exphTu(x)dx 1

The normalization factor Z and the log of it A are defined as:

Z(h)

1,

g (h )

A(h) ln Z (h) ln g(h) ln h( x) exp

h T u( x)

dx

p( x |h) h( x) exp hTu( x) Z (h)

The space of h for which h( x) exp hTu( x) dx < is the natural parameter space.

Bayesian Scientific Computing, Spring 2013 (N. Zabaras)

4

Exponential Family

When the parameter q enters the exponential family as h(q), we write the probability density of the exponential family as follows:

p(x |q ) h(x)g h q exphT q u(x) or

p(x |q ) h(x) exphT q u(x) Ah q ,

where : Ah q log g h q

(q) are the canonical or natural parameters, q is the

parameter vector of some distribution that can be written in the exponential family format

Bayesian Scientific Computing, Spring 2013 (N. Zabaras)

5

 ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download