Download presentation
Presentation is loading. Please wait.
1
CSE 5290: Algorithms for Bioinformatics Fall 2011
Suprakash Datta Office: CSEB 3043 Phone: ext 77875 Course page: 11/11/2018 CSE 5290, Fall 2011
2
Next Clustering revisited: Expectation Maximization, and Gaussian mixture model fitting Some of the following slides are based on slides by Christopher M. Bishop, Microsoft Research, 11/11/2018 CSE 5290, Fall 2011
3
Old Faithful 11/11/2018 CSE 5290, Fall 2011
4
Time between eruptions (minutes)
Old Faithful Data Set Time between eruptions (minutes) Duration of eruption (minutes) 11/11/2018 CSE 5290, Fall 2011
5
K-means Algorithm Goal: represent a data set in terms of K clusters each of which is summarized by a prototype Initialize prototypes, then iterate between two phases: E-step: assign each data point to nearest prototype M-step: update prototypes to be the cluster means Simplest version is based on Euclidean distance re-scale Old Faithful data 11/11/2018 CSE 5290, Fall 2011
6
11/11/2018 CSE 5290, Fall 2011
7
11/11/2018 CSE 5290, Fall 2011
8
11/11/2018 CSE 5290, Fall 2011
9
11/11/2018 CSE 5290, Fall 2011
10
11/11/2018 CSE 5290, Fall 2011
11
11/11/2018 CSE 5290, Fall 2011
12
11/11/2018 CSE 5290, Fall 2011
13
11/11/2018 CSE 5290, Fall 2011
14
11/11/2018 CSE 5290, Fall 2011
15
Responsibilities Responsibilities assign data points to clusters such that Example: 5 data points and 3 clusters 11/11/2018 CSE 5290, Fall 2011
16
K-means Cost Function data prototypes responsibilities 11/11/2018
CSE 5290, Fall 2011
17
Minimizing the Cost Function
E-step: minimize w.r.t. assigns each data point to nearest prototype M-step: minimize w.r.t gives each prototype set to the mean of points in that cluster Convergence guaranteed since there is a finite number of possible settings for the responsibilities 11/11/2018 CSE 5290, Fall 2011
18
Evolution of J 11/11/2018 CSE 5290, Fall 2011
19
Limitations of K-means
Hard assignments of data points to clusters – small shift of a data point can flip it to a different cluster Not clear how to choose the value of K Solution: replace ‘hard’ clustering of K-means with ‘soft’ probabilistic assignments Represents the probability distribution of the data as a Gaussian mixture model 11/11/2018 CSE 5290, Fall 2011
20
The Gaussian Distribution
Multivariate Gaussian Define precision to be the inverse of the covariance In 1-dimension mean covariance 11/11/2018 CSE 5290, Fall 2011
21
Likelihood Function Data set
Assume observed data points generated independently Viewed as a function of the parameters, this is known as the likelihood function 11/11/2018 CSE 5290, Fall 2011
22
Maximum Likelihood Set the parameters by maximizing the likelihood function Equivalently maximize the log likelihood 11/11/2018 CSE 5290, Fall 2011
23
Maximum Likelihood Solution
Maximizing w.r.t. the mean gives the sample mean Maximizing w.r.t covariance gives the sample covariance 11/11/2018 CSE 5290, Fall 2011
24
Bias of Maximum Likelihood
Consider the expectations of the maximum likelihood estimates under the Gaussian distribution The maximum likelihood solution systematically under-estimates the covariance This is an example of over-fitting 11/11/2018 CSE 5290, Fall 2011
25
Intuitive Explanation of Over-fitting
11/11/2018 CSE 5290, Fall 2011
26
Unbiased Variance Estimate
Clearly we can remove the bias by using since this gives Arises naturally in a Bayesian treatment For an infinite data set the two expressions are equal 11/11/2018 CSE 5290, Fall 2011
27
Gaussian Mixtures Linear super-position of Gaussians
Normalization and positivity require Can interpret the mixing coefficients as prior probabilities 11/11/2018 CSE 5290, Fall 2011
28
Example: Mixture of 3 Gaussians
11/11/2018 CSE 5290, Fall 2011
29
Contours of Probability Distribution
11/11/2018 CSE 5290, Fall 2011
30
Sampling from the Gaussian
To generate a data point: first pick one of the components with probability then draw a sample from that component Repeat these two steps for each new data point 11/11/2018 CSE 5290, Fall 2011
31
Synthetic Data Set 11/11/2018 CSE 5290, Fall 2011
32
Fitting the Gaussian Mixture
We wish to invert this process – given the data set, find the corresponding parameters: mixing coefficients means covariances If we knew which component generated each data point, the maximum likelihood solution would involve fitting each component to the corresponding cluster Problem: the data set is unlabelled We shall refer to the labels as latent (= hidden) variables 11/11/2018 CSE 5290, Fall 2011
33
Synthetic Data Set Without Labels
11/11/2018 CSE 5290, Fall 2011
34
Posterior Probabilities
We can think of the mixing coefficients as prior probabilities for the components For a given value of we can evaluate the corresponding posterior probabilities, called responsibilities These are given from Bayes’ theorem by 11/11/2018 CSE 5290, Fall 2011
35
Posterior Probabilities (colour coded)
11/11/2018 CSE 5290, Fall 2011
36
Posterior Probability Map
11/11/2018 CSE 5290, Fall 2011
37
Maximum Likelihood for the GMM
The log likelihood function takes the form Note: sum over components appears inside the log There is no closed form solution for maximum likelihood 11/11/2018 CSE 5290, Fall 2011
38
Over-fitting in GMM Singularities in likelihood function when a component ‘collapses’ onto a data point: then consider Likelihood function gets larger as we add more components (and hence parameters) to the model not clear how to choose the number K of components 11/11/2018 CSE 5290, Fall 2011
39
Problems and Solutions
How to maximize the log likelihood solved by expectation-maximization (EM) algorithm How to avoid singularities in the likelihood function solved by a Bayesian treatment How to choose number K of components also solved by a Bayesian treatment Will not cover Will not cover 11/11/2018 CSE 5290, Fall 2011
40
EM Algorithm – Informal Derivation
Let us proceed by simply differentiating the log likelihood Setting derivative with respect to equal to zero gives giving which is simply the weighted mean of the data 11/11/2018 CSE 5290, Fall 2011
41
EM Algorithm – Informal Derivation II
Similarly for the covariances For mixing coefficients use a Lagrange multiplier to give 11/11/2018 CSE 5290, Fall 2011
42
EM Algorithm – Informal Derivation III
The solutions are not closed form since they are coupled Suggests an iterative scheme for solving them: Make initial guesses for the parameters Alternate between the following two stages: E-step: evaluate responsibilities M-step: update parameters using ML results 11/11/2018 CSE 5290, Fall 2011
43
11/11/2018 CSE 5290, Fall 2011
44
11/11/2018 CSE 5290, Fall 2011
45
11/11/2018 CSE 5290, Fall 2011
46
11/11/2018 CSE 5290, Fall 2011
47
11/11/2018 CSE 5290, Fall 2011
48
11/11/2018 CSE 5290, Fall 2011
49
EM – Latent Variable Viewpoint
Binary latent variables describing which component generated each data point If we knew the values for the LV, we would maximize the complete-data log likelihood which gives a trivial closed-form solution (fit each component to the corresponding set of data points) We don’t know the values of the LV However, for given parameter values we can compute the expected values of the LV 11/11/2018 CSE 5290, Fall 2011
50
Expected Value of Latent Variable
From Bayes’ theorem 11/11/2018 CSE 5290, Fall 2011
51
Complete and Incomplete Data
11/11/2018 CSE 5290, Fall 2011
52
Next A non-bioinformatics topic: Epidemiology
Some of the following slides are based on slides by Dr Bill Hackborn: 11/11/2018 CSE 5290, Fall 2011
53
The problem Build a quantitative model for epidemics
Useful for computer viruses as well Tradeoff between accuracy and tractability 11/11/2018 CSE 5290, Fall 2011
54
What is a Mathematical Model?
a mathematical description of a scenario or situation from the real-world focuses on specific quantitative features of the scenario, ignores others a simplification, abstraction, “cartoon” involves hypotheses that can be tested against real data and refined if desired one purpose is improved understanding of real-world scenario e.g. celestial motion, chemical kinetics 11/11/2018 CSE 5290, Fall 2011
55
Susceptible, Infected, Recovered: the SIR Model of an Epidemic
11/11/2018 CSE 5290, Fall 2011
56
The SIR Epidemic Model First studied, Kermack & McKendrick, 1927.
Consider a disease spread by contact with infected individuals. Individuals recover from the disease and gain further immunity from it. S = fraction of susceptibles in a population I = fraction of infecteds in a population R = fraction of recovereds in a population S + I + R = 1 11/11/2018 CSE 5290, Fall 2011
57
The SIR Epidemic Model II
Differential equations (involving the variables S, I, and R and their rates of change with respect to time t) are An equivalent compartment diagram is 11/11/2018 CSE 5290, Fall 2011
58
Parameters of the Model
r = the infection rate a = the removal rate The basic reproduction number is obtained from these parameters: NR = r /a This number represents the average number of infections caused by one infective in a totally susceptible population. As such, an epidemic can occur only if NR > 1. 11/11/2018 CSE 5290, Fall 2011
59
What does it imply? Typical behaviour 11/11/2018 CSE 5290, Fall 2011
60
Epidemic size Final epidemic size representing the number of nodes that became infectious during the whole epidemic, plotted as a function of rSE in the absence of tracing. Continuous line corresponds to random networks and dashed line to SF networks. Kiss I Z et al. J. R. Soc. Interface 2006;3:55-62
61
Threshold phenomena Prediction of the threshold is critical
Control mechanisms? 11/11/2018 CSE 5290, Fall 2011
62
What does it not model? Heterogeneity Carriers Effect of immunizations
Deaths 11/11/2018 CSE 5290, Fall 2011
63
Vaccination and Herd Immunity
If only a fraction S0 of the population is susceptible, the reproduction number is NRS0, and an epidemic can occur only if this number exceeds 1. Suppose a fraction V of the population is vaccinated against the disease. In this case, S0=1-V and no epidemic can occur if V > 1 – 1/NR The basic reproduction number NR can vary from 3 to 5 for smallpox, 16 to 18 for measles, and over 100 for malaria [Keeling, 2001]. 11/11/2018 CSE 5290, Fall 2011
64
Case Study: Boarding School Flu
11/11/2018 CSE 5290, Fall 2011
65
Boarding School Flu (Cont’d)
time is measured in days, r = 1.66, a = 0.44, and NR = 3.8. 11/11/2018 CSE 5290, Fall 2011
66
Flu at Hypothetical Hospital
In this case, new susceptibles are arriving and those of all classes are leaving. 11/11/2018 CSE 5290, Fall 2011
67
Flu at Hypothetical Hospital II
Parameters r and a are as before. New parameters b = l = 1/14, representing an average turnover time of 14 days. The disease becomes endemic. 11/11/2018 CSE 5290, Fall 2011
68
Case Study: Bombay Plague, 1905-6
The R in SIR often means removed (due to death, quarantine, etc.), not recovered. 11/11/2018 CSE 5290, Fall 2011
69
Enhancing the SIR Model
Can consider additional populations of disease vectors (e.g. fleas, rats). Can consider an exposed (but not yet infected) class, the SEIR model. SIRS, SIS, and double (gendered) models are sometimes used for STDs. Can consider biased mixing, age differences, multiple types of transmission, geographic spread, etc. 11/11/2018 CSE 5290, Fall 2011
70
Why Study Epidemic Models?
To supplement statistical extrapolation. To learn more about the qualitative dynamics of a disease. To test hypotheses about, for example, prevention strategies, disease transmission, significant characteristics, etc. 11/11/2018 CSE 5290, Fall 2011
71
References J. D. Murray, Mathematical Biology, Springer-Verlag, 1989.
O. Diekmann & A. P. Heesterbeek, Mathematical Epidemiology of Infectious Diseases, Wiley, 2000. Matt Keeling, The Mathematics of Diseases, Allyn Jackson, Modeling the Aids Epidemic, Notices of the American Mathematical Society, 36: , 1989. 11/11/2018 CSE 5290, Fall 2011
Similar presentations
