A gentle introduction to Gaussian distribution

Review Random variable Coin flip experiment X = 0X = 1 X: Random variable

Review Probability mass function (discrete) x 01 P(x) P(x) >= 0 Example: Coin flip experiment Any other constraints? Hint: What is the sum?

Review Probability density function (continuous) f(x) x f(x) >= 0 Examples? Unlike discrete, Density function does not represent probability but its rate of change called the “likelihood”

Review Probability density function (continuous) f(x) x f(x) >= 0 x0x0 X 0 +dx P( x 0 < x < x 0 +dx ) = f(x 0 ).dx But, P( x = x 0 ) = 0 & Integrates to 1.0

The Gaussian Distribution Courtesy:

A 2D Gaussian

Central Limit Theorem The distribution of the sum of N i.i.d. random variables becomes increasingly Gaussian as N grows. Example: N uniform [0,1] random variables.

Central Limit Theorem (Coin flip) Flip coin N times Each outcome has an associated random variable X i (=1, if heads, otherwise 0) Number of heads N H is a random variable –Sum of N i.i.d. random variables N H = x 1 + x 2 + …. + x N

Central Limit Theorem (Coin flip) Probability mass function of N H –P(Head) = 0.5 (fair coin) N = 5N = 10N = 40

Geometry of the Multivariate Gaussian

Moments of the Multivariate Gaussian (1) thanks to anti-symmetry of z

Moments of the Multivariate Gaussian (2)

Maximum likelihood Fit a probability density model p(x | θ) to the data –Estimate θ Given independent identically distributed (i.i.d.) data X = (x 1, x 2, …, x N ) –Likelihood –Log likelihood Maximum likelihood: Maximize ln p(X | θ) w.r.t. θ

Maximum Likelihood for the Gaussian (1) Given i.i.d. data, the log likelihood function is given by Sufficient statistics

Maximum Likelihood for the Gaussian (2) Set the derivative of the log likelihood function to zero, and solve to obtain Similarly

Mixtures of Gaussians (1) Old Faithful data set Single GaussianMixture of two Gaussians

Mixtures of Gaussians (2) Combine simple models into a complex model: Component Mixing coefficient K=3

Mixtures of Gaussians (3)

Mixtures of Gaussians (4) Determining parameters ¹, §, and ¼ using maximum log likelihood Solution: use standard, iterative, numeric optimization methods or the expectation maximization algorithm (Chapter 9). Log of a sum; no closed form maximum.

Thank you!