1. LING 696B: Mixture model and linear dimension reduction

2. Statistical estimation
Basic setup:
The world: distributions p(x; ),  -- parameters “all models may be wrong, but some are useful”
Given parameter , p(x; ) tells us how to calculate the probability of x (also referred to as the “likelihood” p(x|) )
Observations: X = {x1, x2, …, xN} generated from some p(x; ). N is the number of observations
Model-fitting: based on some examples X, make guesses (learning, inference) about 

3. Statistical estimation
Example:
Assuming people’s height follows normal distributions (mean, var)
p(x; ) = the probability density function of normal distribution
Observation: measurements of people’s height
Goal: estimate parameters of the normal distribution

4. Maximum likelihood estimate (MLE)
Likelihood function: examples xi are independent of one another, so
Among all the possible values of , choose the so that L() is the biggest
Consistency:
L()
  H ! 

5. H matters a lot!
Example: curve fitting with polynomials

6. Clustering
Need to divide x1, x2, …, xN into clusters, without a priori knowledge of where clusters are
An unsupervised learning problem: fitting a mixture model to x1, x2, …, xN
Example: height of male and female follow two distributions, but don’t know gender from x1, x2, …, xN

7. The K-means algorithm
Start with a random assignment, calculate the means

8. The K-means algorithm
Re-assign members to the closest cluster according to the means

9. The K-means algorithm
Update the means based on the new assignments, and iterate

10. Why does K-means work?
In the beginning, the centers are poorly chosen, so the clusters overlap a lot
But if centers are moving away from each other, then clusters tend to separate better
Vice versa, if clusters are well-separated, then the centers will stay away from each other
Intuitively, these two steps “help each other”

11. Interpreting K-means as statistical estimation
Equivalent to fitting a mixture of Gaussians with:
Spherical covariance
Uniform prior (weights on each Gaussian)
Problems:
Ambiguous data should have gradient membership
Shape of the clusters may not be spherical
Size of the cluster should play a role

12. Multivariate Gaussian
1-D: N(, 2)
N-D: N(, ), ~NX1 vector, ~NXN matrix with (i,j) = ij ~ correlation
Probability calculation:
P(x; ,) = C ||-N/2 exp{-(x-)T -1 (x-)}
Intuitive meaning of -1: how to calculate the distance from x to 
transpose
inverse

13. Multivariate Gaussian: log likelihood and distance
Spherical covariance matrix -1
Diagonal covariance matrix -1
Full covariance matrix -1

14. Learning mixture of Gaussian: EM algorithm
Expectation: putting “soft” labels on data -- a pair (, 1-)
(0.05, 0.95)
(0.8, 0.2)
(0.5, 0.5)

15. Learning mixture of Gaussian: EM algorithm
Maximization: doing Maximum-Likelihood with weighted data
Notice everyone
is wearing a hat!

16. EM v.s. K-means Same: EM better captures the intuition:
Iterative optimization, provably converge (see demo)
EM better captures the intuition:
Ambiguous data are assigned gradient membership
Clusters can be arbitrary shaped pancakes
Size of the cluster is a parameter
Allows for flexible control based on prior knowledge (see demo)

17. EM is everywhere
Our problem: the labels are important, yet not observable – “hidden variables”
This situation is common for complex models, and Maximum likelihood --> EM
Bayesian Networks
Hidden Markov models
Probabilistic Context Free Grammars
Linear Dynamic Systems
Of course, everything we said here still needs to be made more precise if you have to write an algorithm. “guess” has a particular meaning in terms of calculating probabilities, for example. At any rate, this kind of intuition is what is behind the algorithms.
The strategy that was taken in the previous couple of slides has an official name called “EM” algorithm. It has a number of incarnations in computational linguistics. Some of them are quite sophisticated, when you dealing with problems like parsing. But they all follow the same strategy, which is to some extent the best you can do in face of missing information

18. Beyond Maximum likelihood? Statistical parsing
Interesting remark from Mark Johnson:
Intialize a PCFG with treebank counts
Train the PCFG on treebank with EM
A large a mount of NLP research try to dump the first, and improve the second
Measure of success
Log likelihood

19. What’s wrong with this? Mark Johnson’s idea:
Wrong data: human don’t just learn from strings
Wrong model: human syntax isn’t context-free
Wrong way of calculating likelihood: p(sentence | PCFG) isn’t informative
(Maybe) wrong measure of success?

20. End of excursion: Mixture of many things
Any generative model can be combined with a mixture model to deal with categorical data
Examples:
Mixture of Gaussians
Mixture of HMMs
Mixture of Factor Analyzers
Mixture of Expert networks
It all depends on what you are modeling

21. Applying to the speech domain
Speech signals have high dimensions
Using front-end acoustic modeling from speech recognition: Mel-Frequency Cepstral Coefficients (MFCC)
Speech sounds are dynamic
Dynamic acoustic modeling: MFCC-delta
Mixture components are Hidden Markov Models (HMM)
The second challenge is that speech segments are dynamic – they include multiple different events and can be long or short.

22. Clustering speech with K-means
Phones from TIMIT

23. Clustering speech with K-means
Diphones
Words

24. What’s wrong here
Longer sound sequences are more distinguishable for people
Yet doing K-means on static feature vectors misses the change over time
Mixture components must be able to capture dynamic data
Solution: mixture of HMMs

25. Mixture of HMMs HMM HMM Mixture Learning: EM for HMM + EM for mixture
burst
silence
transition
Here each state is supposed to characterize a different portion of speech, and the loops allow each portion to be longer or shorter
Learning: EM for HMM + EM for mixture

26. Mixture of HMMs Model-based clustering Front-end (MFCC+delta)
Algorithm: initial guess by K-means, then EM
Gaussian mixture
for single frames
HMM mixture
for whole sequences

27. Mixture of HMM v.s. K-means
Phone clustering: 7 phones from 22 speakers
They don't fall into 5 manner classes as we wished. But at least you see the improvement
*1 – 5: cluster index

28. Mixture of HMM v.s. K-means
Diphone clustering: 6 diphones from 300+ speakers
If phones are somewhat static, there should be more obvious improvements on more dynamic units. Let's look at diphones

29. Mixture of HMM v.s. K-means
Word clustering: 3 words from 300+ speakers
Distinguishing dynamic sounds are not so difficult for people, if it's hard for the computer then the method must be wrong.

30. Growing the model Guess 6 at once is hard, but 2 is easy;
Hill climbing strategy: starting with 2, then 3, 4, ...
Implementation: split the cluster with the maximum gain in likelihood;
Intuition: discriminate within the biggest pile.

31. Learning categories and features with mixture model
Procedure: apply mixture model and EM algorithm, inductively find clusters
Each split is followed by a retraining step using all data
Data
Here we use a kind of prefix coding to label the clusters, which shows you where each cluster comes from. It is a kind of mnemonic, because the classes arise from the data, and we do not start with assumptions about which sounds should go under which classes. This mnemonic also helps us keep track of things more easily. For example, 112 shows that it comes from 1, then 11, … 2 1 11 12 21 22

32. % classified as Cluster 1 % classified as Cluster 2
IPA TIMIT
All data T D 1
obstruent 2
sonorant  R ? R)
Maybe you can say: what if we draw a line, and cut through here, then we will get the obstruent and sonorants. Actually, what I am trying to demonstrate is, if you consider how to define obstruents and sonorants from a purely acoustic point of view, no such straight line exists! Sonority is a matter of the amount of airflow coming out of you mouth, and you cannot just simply say 1 or 0. What you would expect, instead, is actually a more gradient picture like this one. j l
% classified as Cluster 2

33. % classified as Cluster 12
% classifed as Cluster 11
All data  S 1 1 2 tS T d D 11
fricative 12
Here you might be surprised that affricates do not stand up as a separate category. They kind of stand in between fricatives and stops. But if you look at child speech errors, that seems to be the kind of mistakes that they make – taking affricates as stops or fricatives
% classified as Cluster 12

34. % classified as Cluster 21 % classified as Cluster 22
All data u  r 1 1 2 oU  R oI  A AU 11 12 21
back sonorant 22 aI U  u
In this picture there is another surprise: the liquids – l, w and r go together with the back vowels, and you don’t see a special category of approximants. This is again determined by their spectral property: acoustically, they are much more similar to back vowels than to the other liquid y, which sounds much more like a front high vowel. We are kind of “redefining” phonetic classes only by their acoustic properties, and not worrying about how they are produced.  I i eI i j
% classified as Cluster 22

35. % classified as Cluster 122
% classified Cluster 121
All data 1 1 2 11 12 21 22
121
oral stop
122 nasal stop
% classified as Cluster 122

36. % classified as Cluster 221 % classified as Cluster 222
All data j i 1 1 2  I 11 12 21 22 eI
front
low
sonorant
high
nasal
oral
stop
fricative
back 
121
122
221
222 
% classified as Cluster 222

37. Summary: learning features
Discovered features: distinctions between natural classes based on spectral properties
All data 1
[+sonorant]
[- sonorant]
[+fricative]
[-fricative]
[+back]
[-back]
[-nasal]
[+nasal]
[+high]
[-high]
If features arise from statistical learning, I think it is natural to accept the nature of features as being gradient.
For individual sounds, the feature values are gradient rather than binary (Ladefoged, 01)

38. Evaluation: phone classification
How do the “soft” classes fit into “hard” ones?
Training set
This is just intended to show that our “soft” classes at least make sense.
But it's also worth reflecting what we mean by “errors”.
Test set
Are “errors” really errors?

39. Level 2: Learning segments + phonotactics
Segmentation is a kind of hidden structure
Iterative strategy works here too
Optimization -- the augmented model: p(words | units, phonotactics, segmentation)
Units  argmax p({wi} | U, P, {si}) Clustering = argmax p(segments | units) -- Level 1
Phonotactics  argmax p({wi} | U, P, {si}) Estimating transitions of Markov chain
Segmentation  argmax p({wi} | U, P, {si}) Viterbi decoding
We wanted to do some kind of counting. But we don’t know what to count. Segmentation is an extremely useful things, and that’s what we do when we read spectrograms all the time.
The second aspect of the model is how infants learn segments. As we discussed a couple of slides ago, there are two things that need to be done: one is to find where the segments are, and the other is finding what the segments are.
How can we learn phonetic categories from this sort of data? A critical thing here is “where are the phonetic categories?” So we start with a guess, and use that to update the knowledge. Then we come back and improve the guess, and so on.

40. Iterative learning as coordinate-wise ascent
Level curves
of likelihood score
Units
phonotactics
Initial value
comes from
Level-1 learning
segmentation
Each step increases likelihood score and eventually reaches a local maximum

41. Level 3: Lexicon can be mixtures too
Re-clustering of words using the mixture-based lexical model
Initial values (mixture components, weights)  bottom-up learning (Stage 2)
Iterating steps:
Classify each word as the best exemplar of the given lexical item (also infer segmentation)
Update lexical weights + units + phonotactics

42. Big question: How to choose K?
Basic problem:
Nested hypothesis spaces: Hk-1  Hk  Hk+1  …
As K goes up, likelihood always goes up.
Recall the polynomial curve fitting
Mixture model too (see demo)

43. Big question: How to choose K?
Idea #1: don’t just look at the likelihood, look at the combination of likelihood and something else
Bayesian Information Criterion: -2 log L() + (log N)*d
Minimal Description Length: log L() + description()
Akaike Information Criterion: -2 log L() + 2 d/N
In practice, often need magical “weights” in front of the something else

44. Big question: How to choose K?
Idea #2: use one set of data for learning, one for testing generalization
Cross-validation: run EM until the likelihood starts to hurt in the test set (see demo)
What if you have a bad test set: Jack-knife procedure
Cutting data into 10 parts, and do 10 training and tests

45. Big question: How to choose K?
Idea #3: treat K as “hyper” parameter, and do Bayesian learning on K
More flexible: K can grow up and down depending on number of data
Allow K to grow to infinity: Dirichlet / Chinese restaurant process mixture
Need “hyper-hyper” parameters to control how likely K grows
Computationally also intensive

46. Big question: How to choose K?
There is really no elegant universal solution
One view: statistical learning looks within Hk, but does not come up with Hk itself
How do people choose K? (also see later reading)

47. Dimension reduction Why dimension reduction?
Example: estimate a continuous probability distribution by counting histograms on samples
20 bins
30 bins
10 bins

48. Dimension reduction Now think about 2D, 3D …
How many bins do you need?
Estimate density of distribution with Parzen window:
How big (r) does the window needs to grow?
Data in the window
Window size

49. Curse of dimensionality
Discrete distributions:
Phonetics experiment: M speakers X N sentences X P stresses X Q segments … …
Decision rules: (K) Nearest-neighbor
How big a K is safe?
How long do you have to wait until you are really sure they are your nearest neighbors?

50. One obvious solution Assume we know something about the distribution
Translates to a parametric approach
Example: counting histograms for 10-D data needs lots of bins, but knowing it’s a pancake allows us to fit a Gaussian
d10 parameters v.s. how many?

51. Linear dimension reduction
Principle Components Analysis
Multidimensional Scaling
Factor Analysis
Independent Component Analysis
As we will see, we still need to assume we know something…

52. Principle Component Analysis
Many names (eigen modes, KL transform, etc.) and relatives
The key is to understand how to make a pancake
Centering, rotating and smashing
Step 1: moving the dough to the center
X <-- X - 

53. Principle Component Analysis
Step 2: finding a direction of projection that has the maximal “stretch”
Linear projection of X onto vector w:
Projw(X) = XNXd * wdX1 (X centered)
Now measure the stretch
This is sample variance = Var(X*w) x w

54. Principle Component Analysis
Step 3: formulate this as a constrained optimization problem
Objective of optimization: Var(X*w)
Need constraint on w: (otherwise can explode), only consider the direction
So formally: argmax||w||=1 Var(X*w)

55. Principle Component Analysis
Some algebra (homework): Var(x) = E[(x - E[x]) = E[x2] - (E[x])2
Apply to matrices (homework) Var(X*w) = wTXT * X w = wTCov(X) w (why)
Cov(X) is a dXd matrix (homework)
Symmetric (easy)
For any y, yTCov(X) y >= 0 (tricky)

56. Principle Component Analysis
Going back to the optimization problem: = argmax||w||=1 Var(X*w) = argmax||w||=1 wTCov(X) w
The solution is an eigenvector of Cov(X) w1
The first
Principle Component!

57. More principle components
We keep looking for w2 in all the directions perpendicular to w1
Formally: argmax||w2||=1,w2w1 wTCov(X) w
This turns out to be another eigenvector corresponding to the 2nd largest eigenvalue w2
New coordinates!

58. Rotation
Can keep going until we pick up all d eigenvectors, perpendicular to each other
Putting these eigenvectors together, we have a big matrix W=(w1,w2,…,wd)
W is called an orthogonal matrix
This corresponds to a rotation of the pancake
This pancake has no correlation between dimensions