CSE 5290: Algorithms for Bioinformatics Fall 2011 Suprakash Datta datta@cse.yorku.ca Office: CSEB 3043 Phone: 416-736-2100 ext 77875 Course page: http://www.cse.yorku.ca/course/5290 11/11/2018 CSE 5290, Fall 2011
Next Clustering revisited: Expectation Maximization, and Gaussian mixture model fitting Some of the following slides are based on slides by Christopher M. Bishop, Microsoft Research, http://research.microsoft.com/~cmbishop 11/11/2018 CSE 5290, Fall 2011
Old Faithful 11/11/2018 CSE 5290, Fall 2011
Time between eruptions (minutes) Old Faithful Data Set Time between eruptions (minutes) Duration of eruption (minutes) 11/11/2018 CSE 5290, Fall 2011
K-means Algorithm Goal: represent a data set in terms of K clusters each of which is summarized by a prototype Initialize prototypes, then iterate between two phases: E-step: assign each data point to nearest prototype M-step: update prototypes to be the cluster means Simplest version is based on Euclidean distance re-scale Old Faithful data 11/11/2018 CSE 5290, Fall 2011
11/11/2018 CSE 5290, Fall 2011
11/11/2018 CSE 5290, Fall 2011
11/11/2018 CSE 5290, Fall 2011
11/11/2018 CSE 5290, Fall 2011
11/11/2018 CSE 5290, Fall 2011
11/11/2018 CSE 5290, Fall 2011
11/11/2018 CSE 5290, Fall 2011
11/11/2018 CSE 5290, Fall 2011
11/11/2018 CSE 5290, Fall 2011
Responsibilities Responsibilities assign data points to clusters such that Example: 5 data points and 3 clusters 11/11/2018 CSE 5290, Fall 2011
K-means Cost Function data prototypes responsibilities 11/11/2018 CSE 5290, Fall 2011
Minimizing the Cost Function E-step: minimize w.r.t. assigns each data point to nearest prototype M-step: minimize w.r.t gives each prototype set to the mean of points in that cluster Convergence guaranteed since there is a finite number of possible settings for the responsibilities 11/11/2018 CSE 5290, Fall 2011
Evolution of J 11/11/2018 CSE 5290, Fall 2011
Limitations of K-means Hard assignments of data points to clusters – small shift of a data point can flip it to a different cluster Not clear how to choose the value of K Solution: replace ‘hard’ clustering of K-means with ‘soft’ probabilistic assignments Represents the probability distribution of the data as a Gaussian mixture model 11/11/2018 CSE 5290, Fall 2011
The Gaussian Distribution Multivariate Gaussian Define precision to be the inverse of the covariance In 1-dimension mean covariance 11/11/2018 CSE 5290, Fall 2011
Likelihood Function Data set Assume observed data points generated independently Viewed as a function of the parameters, this is known as the likelihood function 11/11/2018 CSE 5290, Fall 2011
Maximum Likelihood Set the parameters by maximizing the likelihood function Equivalently maximize the log likelihood 11/11/2018 CSE 5290, Fall 2011
Maximum Likelihood Solution Maximizing w.r.t. the mean gives the sample mean Maximizing w.r.t covariance gives the sample covariance 11/11/2018 CSE 5290, Fall 2011
Bias of Maximum Likelihood Consider the expectations of the maximum likelihood estimates under the Gaussian distribution The maximum likelihood solution systematically under-estimates the covariance This is an example of over-fitting 11/11/2018 CSE 5290, Fall 2011
Intuitive Explanation of Over-fitting 11/11/2018 CSE 5290, Fall 2011
Unbiased Variance Estimate Clearly we can remove the bias by using since this gives Arises naturally in a Bayesian treatment For an infinite data set the two expressions are equal 11/11/2018 CSE 5290, Fall 2011
Gaussian Mixtures Linear super-position of Gaussians Normalization and positivity require Can interpret the mixing coefficients as prior probabilities 11/11/2018 CSE 5290, Fall 2011
Example: Mixture of 3 Gaussians 11/11/2018 CSE 5290, Fall 2011
Contours of Probability Distribution 11/11/2018 CSE 5290, Fall 2011
Sampling from the Gaussian To generate a data point: first pick one of the components with probability then draw a sample from that component Repeat these two steps for each new data point 11/11/2018 CSE 5290, Fall 2011
Synthetic Data Set 11/11/2018 CSE 5290, Fall 2011
Fitting the Gaussian Mixture We wish to invert this process – given the data set, find the corresponding parameters: mixing coefficients means covariances If we knew which component generated each data point, the maximum likelihood solution would involve fitting each component to the corresponding cluster Problem: the data set is unlabelled We shall refer to the labels as latent (= hidden) variables 11/11/2018 CSE 5290, Fall 2011
Synthetic Data Set Without Labels 11/11/2018 CSE 5290, Fall 2011
Posterior Probabilities We can think of the mixing coefficients as prior probabilities for the components For a given value of we can evaluate the corresponding posterior probabilities, called responsibilities These are given from Bayes’ theorem by 11/11/2018 CSE 5290, Fall 2011
Posterior Probabilities (colour coded) 11/11/2018 CSE 5290, Fall 2011
Posterior Probability Map 11/11/2018 CSE 5290, Fall 2011
Maximum Likelihood for the GMM The log likelihood function takes the form Note: sum over components appears inside the log There is no closed form solution for maximum likelihood 11/11/2018 CSE 5290, Fall 2011
Over-fitting in GMM Singularities in likelihood function when a component ‘collapses’ onto a data point: then consider Likelihood function gets larger as we add more components (and hence parameters) to the model not clear how to choose the number K of components 11/11/2018 CSE 5290, Fall 2011
Problems and Solutions How to maximize the log likelihood solved by expectation-maximization (EM) algorithm How to avoid singularities in the likelihood function solved by a Bayesian treatment How to choose number K of components also solved by a Bayesian treatment Will not cover Will not cover 11/11/2018 CSE 5290, Fall 2011
EM Algorithm – Informal Derivation Let us proceed by simply differentiating the log likelihood Setting derivative with respect to equal to zero gives giving which is simply the weighted mean of the data 11/11/2018 CSE 5290, Fall 2011
EM Algorithm – Informal Derivation II Similarly for the covariances For mixing coefficients use a Lagrange multiplier to give 11/11/2018 CSE 5290, Fall 2011
EM Algorithm – Informal Derivation III The solutions are not closed form since they are coupled Suggests an iterative scheme for solving them: Make initial guesses for the parameters Alternate between the following two stages: E-step: evaluate responsibilities M-step: update parameters using ML results 11/11/2018 CSE 5290, Fall 2011
11/11/2018 CSE 5290, Fall 2011
11/11/2018 CSE 5290, Fall 2011
11/11/2018 CSE 5290, Fall 2011
11/11/2018 CSE 5290, Fall 2011
11/11/2018 CSE 5290, Fall 2011
11/11/2018 CSE 5290, Fall 2011
EM – Latent Variable Viewpoint Binary latent variables describing which component generated each data point If we knew the values for the LV, we would maximize the complete-data log likelihood which gives a trivial closed-form solution (fit each component to the corresponding set of data points) We don’t know the values of the LV However, for given parameter values we can compute the expected values of the LV 11/11/2018 CSE 5290, Fall 2011
Expected Value of Latent Variable From Bayes’ theorem 11/11/2018 CSE 5290, Fall 2011
Complete and Incomplete Data 11/11/2018 CSE 5290, Fall 2011
Next A non-bioinformatics topic: Epidemiology Some of the following slides are based on slides by Dr Bill Hackborn: http://www.augustana.ualberta.ca/~hackw/mat332/. 11/11/2018 CSE 5290, Fall 2011
The problem Build a quantitative model for epidemics Useful for computer viruses as well Tradeoff between accuracy and tractability 11/11/2018 CSE 5290, Fall 2011
What is a Mathematical Model? a mathematical description of a scenario or situation from the real-world focuses on specific quantitative features of the scenario, ignores others a simplification, abstraction, “cartoon” involves hypotheses that can be tested against real data and refined if desired one purpose is improved understanding of real-world scenario e.g. celestial motion, chemical kinetics 11/11/2018 CSE 5290, Fall 2011
Susceptible, Infected, Recovered: the SIR Model of an Epidemic 11/11/2018 CSE 5290, Fall 2011
The SIR Epidemic Model First studied, Kermack & McKendrick, 1927. Consider a disease spread by contact with infected individuals. Individuals recover from the disease and gain further immunity from it. S = fraction of susceptibles in a population I = fraction of infecteds in a population R = fraction of recovereds in a population S + I + R = 1 11/11/2018 CSE 5290, Fall 2011
The SIR Epidemic Model II Differential equations (involving the variables S, I, and R and their rates of change with respect to time t) are An equivalent compartment diagram is 11/11/2018 CSE 5290, Fall 2011
Parameters of the Model r = the infection rate a = the removal rate The basic reproduction number is obtained from these parameters: NR = r /a This number represents the average number of infections caused by one infective in a totally susceptible population. As such, an epidemic can occur only if NR > 1. 11/11/2018 CSE 5290, Fall 2011
What does it imply? Typical behaviour 11/11/2018 CSE 5290, Fall 2011
Epidemic size Final epidemic size representing the number of nodes that became infectious during the whole epidemic, plotted as a function of rSE in the absence of tracing. Continuous line corresponds to random networks and dashed line to SF networks. Kiss I Z et al. J. R. Soc. Interface 2006;3:55-62
Threshold phenomena Prediction of the threshold is critical Control mechanisms? 11/11/2018 CSE 5290, Fall 2011
What does it not model? Heterogeneity Carriers Effect of immunizations Deaths 11/11/2018 CSE 5290, Fall 2011
Vaccination and Herd Immunity If only a fraction S0 of the population is susceptible, the reproduction number is NRS0, and an epidemic can occur only if this number exceeds 1. Suppose a fraction V of the population is vaccinated against the disease. In this case, S0=1-V and no epidemic can occur if V > 1 – 1/NR The basic reproduction number NR can vary from 3 to 5 for smallpox, 16 to 18 for measles, and over 100 for malaria [Keeling, 2001]. 11/11/2018 CSE 5290, Fall 2011
Case Study: Boarding School Flu 11/11/2018 CSE 5290, Fall 2011
Boarding School Flu (Cont’d) time is measured in days, r = 1.66, a = 0.44, and NR = 3.8. 11/11/2018 CSE 5290, Fall 2011
Flu at Hypothetical Hospital In this case, new susceptibles are arriving and those of all classes are leaving. 11/11/2018 CSE 5290, Fall 2011
Flu at Hypothetical Hospital II Parameters r and a are as before. New parameters b = l = 1/14, representing an average turnover time of 14 days. The disease becomes endemic. 11/11/2018 CSE 5290, Fall 2011
Case Study: Bombay Plague, 1905-6 The R in SIR often means removed (due to death, quarantine, etc.), not recovered. 11/11/2018 CSE 5290, Fall 2011
Enhancing the SIR Model Can consider additional populations of disease vectors (e.g. fleas, rats). Can consider an exposed (but not yet infected) class, the SEIR model. SIRS, SIS, and double (gendered) models are sometimes used for STDs. Can consider biased mixing, age differences, multiple types of transmission, geographic spread, etc. 11/11/2018 CSE 5290, Fall 2011
Why Study Epidemic Models? To supplement statistical extrapolation. To learn more about the qualitative dynamics of a disease. To test hypotheses about, for example, prevention strategies, disease transmission, significant characteristics, etc. 11/11/2018 CSE 5290, Fall 2011
References J. D. Murray, Mathematical Biology, Springer-Verlag, 1989. O. Diekmann & A. P. Heesterbeek, Mathematical Epidemiology of Infectious Diseases, Wiley, 2000. Matt Keeling, The Mathematics of Diseases, http://plus.maths.org, 2004. Allyn Jackson, Modeling the Aids Epidemic, Notices of the American Mathematical Society, 36:981-983, 1989. 11/11/2018 CSE 5290, Fall 2011