1. Maximum Likelihood Estimation of Mixture Densities for Binned and Truncated Multivariate Data
Igor V. Cadez, Padhraic Smyth, Geoff J. Mclachlan, Christine and E. McLaren, Machine Learning 2001 (to appear)
O, Jangmin
2001/06/01

2. Introduction (1)
Fitting mixture models to binned and truncated data by ML via EM.
Binning
measurement with finite resolution
quantifying real-valued variables
Truncation
Motivation
diagnostic evaluation of anemia
volume of RBC, amount of hemoglobin : measured by cytometric blood cell counter (Bayer Corp.)

3. Figure 1

4. Introduction (2) Data in the form of histogram Binning Truncation
Computer Vision, Massive data sets, …
Binning
Measurement Precision
Truncation
Limitation of the range of measurement, intentionally, …
EM frame work
Missing data: original data points.

5. Binned and Truncated Data
Sample space
v mutually exclusive regions Hr (r=1,…,v)
Observation
Only the number of nr of the Yj that fall in Hr (r=1,…,v0) is recorded (v0  v).
Observed data vector :
a is multinomial distribution

6. Observed log likelihood

7. Application of EM Algorithm : Missing Data
Unobservable frequencies in the case of truncation.
nr unobservable individuals in the rth region Yr.
Complete Data vector

8. p(u|a;) can be specified… (negative binomial ?)
p(a;) is specified
p(u|a;) can be specified… (negative binomial ?)
p(y1+,…, yv+|u, a; ) is specified
Conditioning on u and a, yj+ is composed by independent nj sampling from the density

9. Application of EM Algorithm : Missing Data
Then, complete data log-likelihood

10. Application of EM Algorithm : Mixture Model
Extension to mixture model (g components)
Conditional probability that Yrs belongs to i-th component given yrs.
Final complete data log-likelihood
Zero-one indicator variable

11. E-Step Calculation of Q(; (k)) expection over y1+,…,yv+
expection over u .
Expectation of u given a …

12. M-Step
i(k+1) update
 = (1,…, g) : other parameters are adjusted to be…

13. M-Step for Normal Components
Parameter update equation
Practical implementation is more complex due to multinomial integrals.

14. Computational and Numerical Issues
Integration can’t be evaluated analytically.
m bins in univariate, O(md) in d-dimensional.
O(i) evaluation in univariate integration, O(id) in d-dimensional
Complex geometry.
For fixed sample size, more sparser multivariate histogram
Integrating methods
Numerical
Monte Carlo
Romberg : Idea – repeated 1-dimensional integration.

15. Handling Truncated Regions
A single bin
No extra integration is needed.

16. 3.3 The Complete EM Algorithm
Treat the histogram as a PDF and draw a small number of data points from it
Fit the mixture model using the standard EM algorithm (nonbinned , nontruncated)
Using the parameter estimates from above, refine the estimate with the full EM algorithm applied to the binned and truncated data

17. 4. Experimental Results with Simulated Data
3 experiments
Generate data from a known PDF and then bin them (bivariate).
Number of bin per dimension: 5 ~ 100 (step 5)
10 different samples for smoothing results.
Standard EM on unbinned samples v.s. full EM on binned samples
Estimation method: KL distance between true density v.s. 2 EMs

18. Experiment Setup
To test the quality of the solution for different numbers of data points from Figure 4.
Data points N : 100 ~ 1000 (step 10)
(20 bin, 100 data, 10 samples)
To test performance of the algorithm when the component densities are not so well separated.
3 apart components
(20 bin, 20 separation, 10 samples)
To test the performance of the algorithm when significant truncation occurs
(20 bin, 100 positions, 10 samples)

21. 4.2 Estimation from Random Samples Generated from the Binned Data
Baseline approach
Estimate PDF from a random sample from the binned data
Uniform sampling estimation method
Figure 6 : comparison
Overestimates the variance
Variance inflation

22. Figure 6 : Estimated PDFs obtained from original data and PDFs fitted by binned and the uniform random-sample algorithm for (a) 5 bins per dimension and (b) 10 per dimension. 3-covariance ellipse

23. 4.3 Experiments with Different Sample Size
Figure 7
As a function of number of bins and number of data points
Bin > 20, data > 500 : small KL distance
Figure 8
As a function of number of bins
Bin (5 ~ 20): rapid decay, Bin > 20 : flat
Figure 9
As a function of number of data
Exponential decay

24. Figure 7 : (a) average KL distance between the estimated density and the true density, (b) standard deviation of the KL distance from10 repeated samples.

27. 4.4 Experiments with Different Separations of Mixture Components
Figure 10
As a function of number of bins and separation of mean
Insensitive to separation of components
Figure 11
As a function of separation of mean
Ratio of KL distance of the standard and binned algorithm
Small number of bin : standard EM is better.
Small separation : binned EM is better
Figure 12

31. 4.5 Experiments with Truncation
Figure 13
Function of ratio of truncated points
Standard EM ignores the information of truncation
Relatively insensitive to truncation, in binned EM
Figure 14

34. Real Example : Red Blood Cell Data
Medical diagnosis
based on two-dimensional histograms characterizing RBC and hemoglobin measurements
Mixture densities were fitted to histograms from 90 control subject and 82 subjects with iron deficient anemia
B=1002, N=40,000
Using for discriminant rule
Baseline features: 4-dim feature vector (mean, variance along RBC and hemoglobin)
11-dim features: two-component lognormal mixture models (mean, cov, mixing weight)
9-dim features: (mean, log-odds of eigenvalues of cov, mixing weight)

35. Figure 15. Contour plots from estimated density estimates for three control patients and three iron deficient anemia patients.

37. Conclusion
Fitting mixture densities to multivariate binned and truncated data
Computational and numerical implementation issues
In 2-dim simulation, If number of bins exceeds 10 the loss of information from quantization is minimal.