1. Bayesian Rule & Gaussian Mixture Models
Jianping Fan
Dept of Computer Science
UNC-Charlotte

2. Input Output Basic Classification Spam vs. Not-Spam Spam Binary
filtering
Binary
!!!!$$$!!!!

3. C Input Output Basic Classification Multi-Class Character recognition
C vs. other 25 characters

4. Structured Classification
Input
Output
Handwriting
recognition
Structured output
Graph Model
brace
building
3D object
recognition
tree

5. Overview of Bayesian Decision
Bayesian classification: one example
E.g. How to decide if a patient is sick or healthy, based on
A probabilistic model of the observed data (data distributions)
Prior knowledge (ratio or importance)
Majority has bigger voice

6. Overview of Bayesian Decision
Test patient
Group assignment for test patient;
Prior knowledge about the assigned group
(c) Properties of the assigned group (sick or healthy)

7. Overview of Bayesian Decision
Test patient
Observations: Bayesian decision process is a data modeling
process, e.g., estimate the data distribution
K-means clustering: any relationship & difference?

8. Bayes’ Rule:
normalization
Who is who in Bayes’ rule

9. Classification problem
Training data: examples of the form (d,h(d))
where d are the data objects to classify (inputs)
and h(d) are the correct class info for d, h(d){1,…K}
Goal: given dnew, provide h(dnew)

10. Why Bayesian? Provides practical learning algorithms E.g. Naïve Bayes
Prior knowledge and observed data can be combined
It is a generative (model based) approach, which offers a useful conceptual framework
E.g. sequences could also be classified, based on a probabilistic model specification
Any kind of objects can be classified, based on a probabilistic model specification

11. Bayes’ Rule: P(d|h): data distribution of group h P(h): importance
data distribution of whole
data set

12. Bayes’ Rule: Estimating the data distribution for whole data set
Works to support Bayesian decision:
Estimating the data distribution for whole data set
Estimating the data distribution for specific group
Prior knowledge about specific group

13. Gaussian Mixture Model (GMM)
How to estimate the data distribution for a given dataset?

14. Gaussian Mixture Model (GMM)

15. Gaussian Mixture Model (GMM)
What does GMM mean?
100 = 5*10 + 2*20 + 2*5
100 = 100*1
100 = 10*10
……………….
GMM may prefer larger K with ``smaller” Gaussians

16. Gaussian Mixture Model (GMM)

17. Gaussian Mixture Model (GMM)
Real Data Distribution

18. Gaussian Mixture Model (GMM)
Real Data Distribution
Approximated Gaussian functions

19. Gaussian Mixture Model (GMM)
Parameters to be estimated:
Training Set:
We assume K is available (or pre-defined)! Algorithms
may always prefer larger K!

20. Matching between Gaussian Function and Samples 
Sampling

21. Given x, it is a function of  and 2
Maximum Likelihood  
Sampling
We want to maximize it.
Given x, it is a function of  and 2

22. Log-Likelihood Function
Maximize
this instead
By setting
and

23. Max. the Log-Likelihood Function

24. Max. the Log-Likelihood Function

25. Gaussian Mixture Models
Rather than identifying clusters by “nearest” centroids
Fit a Set of k Gaussians to the data
Maximum Likelihood over a mixture model
p(x) = \pi_0f_0(x) + \pi_1f_1(x) + \pi_2f_2(x) + \ldots + \pi_kf_k(x)

26. GMM example

27. Mixture Models
Formally a Mixture Model is the weighted sum of a number of pdfs where the weights are determined by a distribution,

28. Gaussian Mixture Models
GMM: the weighted sum of a number of Gaussians where the weights are determined by a distribution,

29. Expectation Maximization
Both the training of GMMs and Graphical Models with latent variables can be accomplished using Expectation Maximization
Step 1: Expectation (E-step)
Evaluate the “responsibilities” of each cluster with the current parameters
Step 2: Maximization (M-step)
Re-estimate parameters using the existing “responsibilities”
Similar to k-means training.

30. Latent Variable Representation
We can represent a GMM involving a latent variable
What does this give us?
TODO: plate notation

31. GMM data and Latent variables

32. One last bit
We have representations of the joint p(x,z) and the marginal, p(x)…
The conditional of p(z|x) can be derived using Bayes rule.
The responsibility that a mixture component takes for explaining an observation x.

33. Maximum Likelihood over a GMM
As usual: Identify a likelihood function
And set partials to zero…

34. Maximum Likelihood of a GMM
Optimization of means.

35. Maximum Likelihood of a GMM
Optimization of covariance
Note the similarity to the regular MLE without responsibility terms.

36. Maximum Likelihood of a GMM
Optimization of mixing term
\frac{\partial \ln p(x|\pi, \mu,\Sigma) + \lambda\left(\sum_{k=1}^K \pi_k -1\right)}{\partial \pi_k}&=&

37. MLE of a GMM

38. EM for GMMs Initialize the parameters
Evaluate the log likelihood
Expectation-step: Evaluate the responsibilities
Maximization-step: Re-estimate Parameters
Check for convergence

39. EM for GMMs
E-step: Evaluate the Responsibilities

40. EM for GMMs
M-Step: Re-estimate Parameters

41. EM for GMMs Evaluate the log likelihood
and check for convergence of either the parameters or the log likelihood. If the convergence criterion is not satisfied, return to E-Step.

42. Visual example of EM

43. Incorrect Number of Gaussians

44. Incorrect Number of Gaussians

45. Too many parameters
For n-dimensional data, we need to estimate at least 3n parameters

46. Relationship to K-means
K-means makes hard decisions.
Each data point gets assigned to a single cluster.
GMM/EM makes soft decisions.
Each data point can yield a posterior p(z|x)
Soft K-means is a special case of EM.

47. Soft means as GMM/EM
Assume equal covariance matrices for every mixture component:
Likelihood:
Responsibilities:
As epsilon approaches zero, the responsibility approaches unity.
p(x|\mu_k,\Sigma_k) = \frac{1}{(2\pi\epsilon)^{M/2}}\exp\left\{-\frac{1}{2\epsilon}\lVert x-\mu_k\rVert^2\right\}

48. Soft K-Means as GMM/EM
Overall Log likelihood as epsilon approaches zero:
The expectation of soft k-means is the inter-cluster variability
Note: only the means are re-estimated in Soft K- means.
The covariance matrices are all tied.

49. General form of EM
Given a joint distribution over observed and latent variables:
Want to maximize:
Initialize parameters
E Step: Evaluate:
M-Step: Re-estimate parameters (based on expectation of complete-data log likelihood)
Check for convergence of parameters or likelihood

50. Gaussian Mixture Model

51. The conditional probability p(zk = 1|x) denoted by γ(zk ) is obtained by Bayes' theorem,
We view πk as the prior probability of zk = 1, and γ(zk ) as the posterior probability.
γ(zk ) is the responsibility that k-component takes for explaining the observation x.

52. Maximum Likelihood Estimation
Given a data set of X = {x1,…, xN}, the log of the likelihood function is
s.t.

53. Setting the derivatives of ln p(X|π, μ, Σ) with respect to μk to zero, we obtain

54. Finally, we maximize ln p(X|π, μ, Σ) with respect to the mixing coefficients πk. We use a Largrange multiplier
objective ftn. f(x)
constraints g(x)
max f(x) s.t. g(x)=0

55. which gives
we find λ = -N and

56. Initialise the means μk , covariances Σk and mixing coefficients πk.
Ε step: Evaluate the responsibilities using the current parameter values
3. M step: RE-estimate the parameters using the current responsibilities
4. Evaluate the log likelihood
and check for convergence of either the parameters or the log likelihood. If the convergence criterion is not satisfied, return to step 2.

58. Source code for GMM 1. EPFL http://lasa.epfl.ch/sourcecode/
2. Google Source Code on GMM
3. GMM & EM

59. Probabilities – auxiliary slide for memory refreshing
Have two dice h1 and h2
The probability of rolling an i given die h1 is denoted P(i|h1). This is a conditional probability
Pick a die at random with probability P(hj), j=1 or 2. The probability for picking die hj and rolling an i with it is called joint probability and is P(i, hj)=P(hj)P(i| hj).
For any events X and Y, P(X,Y)=P(X|Y)P(Y)
If we know P(X,Y), then the so-called marginal probability P(X) can be computed as

60. Does patient have cancer or not?
A patient takes a lab test and the result comes back positive. It is known that the test returns a correct positive result in only 98% of the cases and a correct negative result in only 97% of the cases. Furthermore, only of the entire population has this disease.
1. What is the probability that this patient has cancer?
2. What is the probability that he does not have cancer?
3. What is the diagnosis?

62. Choosing Hypotheses Maximum Likelihood hypothesis:
Generally we want the most probable hypothesis given training data. This is the maximum a posteriori hypothesis:
Useful observation: it does not depend on the denominator P(d)

63. Now we compute the diagnosis
To find the Maximum Likelihood hypothesis, we evaluate P(d|h) for the data d, which is the positive lab test and chose the hypothesis (diagnosis) that maximises it:
To find the Maximum A Posteriori hypothesis, we evaluate P(d|h)P(h) for the data d, which is the positive lab test and chose the hypothesis (diagnosis) that maximises it. This is the same as choosing the hypotheses gives the higher posterior probability.

64. Naïve Bayes Classifier
What can we do if our data d has several attributes?
Naïve Bayes assumption: Attributes that describe data instances are conditionally independent given the classification hypothesis
it is a simplifying assumption, obviously it may be violated in reality
in spite of that, it works well in practice
The Bayesian classifier that uses the Naïve Bayes assumption and computes the MAP hypothesis is called Naïve Bayes classifier
One of the most practical learning methods
Successful applications:
Medical Diagnosis
Text classification

65. Example. ‘Play Tennis’ data

66. Naïve Bayes solution Classify any new datum instance x=(a1,…aT) as:
To do this based on training examples, we need to estimate the parameters from the training examples:
For each target value (hypothesis) h
For each attribute value at of each datum instance

67. Based on the examples in the table, classify the following datum x:
x=(Outl=Sunny, Temp=Cool, Hum=High, Wind=strong)
That means: Play tennis or not?
Working:

68. Learning to classify text
Learn from examples which articles are of interest
The attributes are the words
Observe the Naïve Bayes assumption just means that we have a random sequence model within each class!
NB classifiers are one of the most effective for this task
Resources for those interested:
Tom Mitchell: Machine Learning (book) Chapter 6.

69. Results on a benchmark text corpus

70. Remember Bayes’ rule can be turned into a classifier
Maximum A Posteriori (MAP) hypothesis estimation incorporates prior knowledge; Max Likelihood doesn’t
Naive Bayes Classifier is a simple but effective Bayesian classifier for vector data (i.e. data with several attributes) that assumes that attributes are independent given the class.
Bayesian classification is a generative approach to classification

71. Resources
Textbook reading (contains details about using Naïve Bayes for text classification):
Tom Mitchell, Machine Learning (book), Chapter 6.
Software: NB for classifying text:
Useful reading for those interested to learn more about NB classification, beyond the scope of this module:

72. GMM & Bayesian Decision
Open Issues
Number of Gaussians K
Initializations of parameters
Pay more attentions on majority

73. GMM & Bayesian Rule Advantages Intuitive modeling Probabilistic
Disadvantages
Intuitive modeling
Probabilistic
Multi-class solution
Number of Gaussians K
Initializations of Parameters
Not-discriminative
Feature dimensions
Majority vs. minority

74. GMM for K-Means Clustering

75. GMM for K-Means Clustering

76. KNN (K nearest neighbors) Classifier
Test sample

77. Problems of GMM Number of Gaussians K How to estimate K?
Majority vs. Minority
How to make right decision for rare events?

78. Problems of GMM Parameter Initialization & Feature Dimensions
What are the solutions?
Not-discriminative
What’s the solution?