Unsupervised Learning: Clustering Some material adapted from slides by Andrew Moore, CMU. Visit for Andrew’s repository of Data Mining tutorials.

Unsupervised Learning Supervised learning used labeled data pairs (x, y) to learn a function f : X→Y. Supervised learning used labeled data pairs (x, y) to learn a function f : X→Y. But, what if we don’t have labels? But, what if we don’t have labels? No labels = unsupervised learning No labels = unsupervised learning Only some points are labeled = semi-supervised learning Only some points are labeled = semi-supervised learning Labels may be expensive to obtain, so we only get a few. Labels may be expensive to obtain, so we only get a few. Clustering is the unsupervised grouping of data points. It can be used for knowledge discovery. Clustering is the unsupervised grouping of data points. It can be used for knowledge discovery.

Clustering Data

K-Means Clustering K-Means ( k, data ) Randomly choose k cluster center locations (centroids). Loop until convergence Assign each point to the cluster of the closest centroid. Reestimate the cluster centroids based on the data assigned to each.

K-Means Clustering K-Means ( k, data ) Randomly choose k cluster center locations (centroids). Loop until convergence Assign each point to the cluster of the closest centroid. Reestimate the cluster centroids based on the data assigned to each.

K-Means Clustering K-Means ( k, data ) Randomly choose k cluster center locations (centroids). Loop until convergence Assign each point to the cluster of the closest centroid. Reestimate the cluster centroids based on the data assigned to each.

K-Means Animation Example generated by Andrew Moore using Dan Pelleg’s super- duper fast K-means system: Dan Pelleg and Andrew Moore. Accelerating Exact k-means Algorithms with Geometric Reasoning. Proc. Conference on Knowledge Discovery in Databases 1999.

Problems with K-Means Very sensitive to the initial points. Very sensitive to the initial points. Do many runs of k-Means, each with different initial centroids. Do many runs of k-Means, each with different initial centroids. Seed the centroids using a better method than random. (e.g. Farthest-first sampling) Seed the centroids using a better method than random. (e.g. Farthest-first sampling) Must manually choose k. Must manually choose k. Learn the optimal k for the clustering. (Note that this requires a performance measure.) Learn the optimal k for the clustering. (Note that this requires a performance measure.)

Problems with K-Means How do you tell it which clustering you want? How do you tell it which clustering you want? Constrained clustering techniques Constrained clustering techniques Same-cluster constraint (must-link) Different-cluster constraint (cannot-link)

Learning Bayes Nets Some material adapted from lecture notes by Lise Getoor and Ron Parr Adapted from slides by Tim Finin and Marie desJardins.

Learning Bayesian networks Given training set Given training set Find B that best matches D Find B that best matches D model selection model selection parameter estimation parameter estimation Data D Inducer C A EB

Parameter estimation Assume known structure Assume known structure Goal: estimate BN parameters  Goal: estimate BN parameters  entries in local probability models, P(X | Parents(X)) entries in local probability models, P(X | Parents(X)) A parameterization  is good if it is likely to generate the observed data: A parameterization  is good if it is likely to generate the observed data: Maximum Likelihood Estimation (MLE) Principle: Choose   so as to maximize L Maximum Likelihood Estimation (MLE) Principle: Choose   so as to maximize L i.i.d. samples

Parameter estimation II The likelihood decomposes according to the structure of the network The likelihood decomposes according to the structure of the network → we get a separate estimation task for each parameter The MLE (maximum likelihood estimate) solution: The MLE (maximum likelihood estimate) solution: for each value x of a node X for each value x of a node X and each instantiation u of Parents(X) and each instantiation u of Parents(X) Just need to collect the counts for every combination of parents and children observed in the data Just need to collect the counts for every combination of parents and children observed in the data MLE is equivalent to an assumption of a uniform prior over parameter values MLE is equivalent to an assumption of a uniform prior over parameter values sufficient statistics

Sufficient statistics: Example Why are the counts sufficient? Why are the counts sufficient? EarthquakeBurglary Alarm Moon-phase Light-level θ * A | E, B = N(A, E, B) / N(E, B)

Model selection Goal: Select the best network structure, given the data Input: Training data Training data Scoring function Scoring functionOutput: A network that maximizes the score A network that maximizes the score

Structure selection: Scoring Bayesian: prior over parameters and structure Bayesian: prior over parameters and structure get balance between model complexity and fit to data as a byproduct get balance between model complexity and fit to data as a byproduct Score (G:D) = log P(G|D)  log [P(D|G) P(G)] Score (G:D) = log P(G|D)  log [P(D|G) P(G)] Marginal likelihood just comes from our parameter estimates Marginal likelihood just comes from our parameter estimates Prior on structure can be any measure we want; typically a function of the network complexity Prior on structure can be any measure we want; typically a function of the network complexity Same key property: Decomposability Score(structure) =  i Score(family of X i ) Marginal likelihood Prior

Heuristic search B E A C B E A C B E A C B E A C Δscore(C) Add E  C Δscore(A) Delete E  A Δscore(A) Reverse E  A

Exploiting decomposability B E A C B E A C B E A C Δscore(C) Add E  C Δscore(A) Delete E  A Δscore(A) Reverse E  A B E A C Δscore(A) Delete E  A To recompute scores, only need to re-score families that changed in the last move

Variations on a theme Known structure, fully observable: only need to do parameter estimation Known structure, fully observable: only need to do parameter estimation Unknown structure, fully observable: do heuristic search through structure space, then parameter estimation Unknown structure, fully observable: do heuristic search through structure space, then parameter estimation Known structure, missing values: use expectation maximization (EM) to estimate parameters Known structure, missing values: use expectation maximization (EM) to estimate parameters Known structure, hidden variables: apply adaptive probabilistic network (APN) techniques Known structure, hidden variables: apply adaptive probabilistic network (APN) techniques Unknown structure, hidden variables: too hard to solve! Unknown structure, hidden variables: too hard to solve!

Handling missing data Suppose that in some cases, we observe earthquake, alarm, light-level, and moon-phase, but not burglary Suppose that in some cases, we observe earthquake, alarm, light-level, and moon-phase, but not burglary Should we throw that data away?? Should we throw that data away?? Idea: Guess the missing values based on the other data Idea: Guess the missing values based on the other data EarthquakeBurglary Alarm Moon-phase Light-level

EM (expectation maximization) Guess probabilities for nodes with missing values (e.g., based on other observations) Guess probabilities for nodes with missing values (e.g., based on other observations) Compute the probability distribution over the missing values, given our guess Compute the probability distribution over the missing values, given our guess Update the probabilities based on the guessed values Update the probabilities based on the guessed values Repeat until convergence Repeat until convergence

EM example Suppose we have observed Earthquake and Alarm but not Burglary for an observation on November 27 Suppose we have observed Earthquake and Alarm but not Burglary for an observation on November 27 We estimate the CPTs based on the rest of the data We estimate the CPTs based on the rest of the data We then estimate P(Burglary) for November 27 from those CPTs We then estimate P(Burglary) for November 27 from those CPTs Now we recompute the CPTs as if that estimated value had been observed Now we recompute the CPTs as if that estimated value had been observed Repeat until convergence! Repeat until convergence! EarthquakeBurglary Alarm