Unsupervised Group Discovery in Relational Datasets: A nonparametric Bayesian Approach P.S. Koutsourelakis School of Civil and Environmental Engineering Cornell University Artificial Intelligence Seminar, 10/12/07 Joint work with T. Eliassi-Rad, LLNL

P.S. Koutsourelakis, Problem Setting A B D C age income location … age income location … age income location … age income location … friend co-worker phone call Traditional Clustering  Can we improve clustering by using relational data ?  What if only relational data was available ?  Can we make predictions about missing links or attributes?

P.S. Koutsourelakis, Problem Setting A collection of objects belonging to various types/domains (i.e. people, papers, locations, devices, movies, etc) Each object might have (observable) attributes Links/relations between: – Two or more objects – Objects can be of the same or different types – Binary (absence/presence), integer or real-valued Each link might have (observable) attributes Find groups of objects of each type, or Find common identities between objects of each type, or Organize objects into clusters that relate to each other in predictable ways Goal:

P.S. Koutsourelakis, Problem Setting A B D C Given an adjacency matrix where R i,j = 0 or 1 (observables), find cluster assignment I i (hidden/latent). ABCD A 000 B 0 00 C 10 0 D 011 Probabilistic - Bayesian Formulation posterior likelihood prior

P.S. Koutsourelakis, Problem Setting Likelihood: The relational behavior of the objects is completely determined by their cluster assignments I i For example: matrix specifying link probability between any two groups Kemp, C., Tenenbaum, J. B., Griffiths, T. L., Yamada, T. & Ueda, N. Learning systems of concepts with an infinite relational model. AAAI 2006.

P.S. Koutsourelakis, Augmented Problem Setting If objects have attributes (i.e., x i which are also observed), then we can augment likelihood : If links R i,j are real-valued (i.e. duration of phone call, number of bytes etc): Functions of group assignments

P.S. Koutsourelakis, Problem Setting We need a prior on group assignments p(I). What is an appropriate prior p(K) on the number of clusters K? Groups are unlikely to be related as above. The distribution on I i should be exchangeable. That is, the order in which nodes are assigned can be permuted without changing the probability of resulting partition. Likelihood Function

P.S. Koutsourelakis, Nonparametric Bayesian Methods* Bayesian methods are most powerful when your prior adequately captures your beliefs. Inflexible models (e.g. with a fixed number of groups) might yield unreasonable inferences. Non-parametrics provide a way of getting very flexible models. Non-parametric models can automatically infer an adequate model size/complexity from the data, without needing to explicitly do Bayesian model comparison Many can be derived by starting with a finite parametric model and taking the limit as number of parameters * Nonparametric doesn’t mean there are no parameters, but that “the number of parameters grows with the data” (e.g. as in Parzen window density estimation)

P.S. Koutsourelakis, Chinese Restaurant Process (CRP) (potentially infinite dishes) MENU.

P.S. Koutsourelakis, Chinese Restaurant Process (CRP) Properties: CRP is exchangeable (i.e. order in which customers entered doesn’t matter) The number of groups grows as O(log n) where n is the number of nodes Inference with Gibbs sampling can be based on the conditionals above Larger γ favors more clusters number of people already eating dish j

P.S. Koutsourelakis, Infinite Relational Model (IRM) “Forward” Interpretation (single domain) 1) Sample group assignments I i from CRP(γ) resulting in K clusters 2) Sample iid η(a,b) for all a,b=1,2,..,K from Beta(β 1,β 2 ) 3) Sample iid each R i,j from Bernoulli(η(I i, I j )) From Kemp, C., Tenenbaum, J. B., Griffiths, T. L., Yamada, T. & Ueda, N. Learning systems of concepts with an infinite relational model. AAAI 2006.

P.S. Koutsourelakis, 2 domains (animals + features) Animals form two groups: birds + 4-legged mammals Application: Object-Feature Dataset

P.S. Koutsourelakis, Application: Object-Feature Dataset Maximum-Likelihood Configuration Animal Domain Group 1: dove, hen, owl, falcon, eagle Group 2: duck, goose Group 3: fox, cat Group 4: horse, zebra Group 5: dog, wolf, tiger, lion, cow Feature Domain Group 1: small, 2-legs, feathers, fly Group 2: medium, hunt Group 3: big, hooves, mane, run Group 4: 4-legs, hair Group 5: swim

P.S. Koutsourelakis, Application: Object-Feature Dataset

P.S. Koutsourelakis, Predicting Missing Links % of Missing Links AUCAccuracy 10% % % % Can we make predictions about missing links?

P.S. Koutsourelakis, Infinite Relational Model (IRM) Advantages: It is an unsupervised learner with only two tunable parameters β and γ. It can be applied to multiple node types and relations. It has all the advantages of a Bayesian formulations (missing data, confidence intervals) and nonparametric methods (adaptation to data, outlier accommodation). It has been successfully used for co-clustering object features, learning ontologies and social networks. Disadvantages: Significant computational effort It does not capture “multiple personalities.”

P.S. Koutsourelakis, “Multiple Personalities” In real data, objects (e.g. people) do not belong exclusively to one group, i.e. their identity is a mixture of basic components. These components can be the same for each object type but the mixing proportions might vary from one object to another.. IRM assumes that each object participates in all the relations it is involved with a single identity. A proper model should account for a different mixture for each object over all the possible identity components (which are common for the whole domain). This way we learn not only all the groups of the population but also all the existing mixtures of them. This can be achieved by introducing a Bayesian hierarchy groups ≡ identities

P.S. Koutsourelakis, Mixed-Membership Model (MMM) A: No, because the groups for each CRP will not be shared across objects Q: Can we use an independent CRP for each object

P.S. Koutsourelakis, Chinese Restaurant Franchise N restaurants with a common menu Object 1 = restaurant 1 Object 2 = restaurant 2 Object N = restaurant N ……………… Phase 1: Table Assignment Phase 2: Dish Assignment Y.W. Teh, M.I. Jordan, M.J. Beal and D.M. Blei. Hierarchical Dirichlet Processes. JASA, 2006.

P.S. Koutsourelakis, Chinese Restaurant Franchise number of customers already sitting at table t table assignment for customer m at restaurant i number of tables number of tables already eating dish k dish assignment for table t in restaurant i

P.S. Koutsourelakis, Mixed-Membership Model dish assignment of node i Properties: - Has a few more parameters, γ i, but also has higher expressivity - Inference with Gibbs sampling can be based on the conditionals above

P.S. Koutsourelakis, Non-Identifiability A B two objects: A, B two groups: 1, 2 100% group 1 50% group 1 50% group 2 Probability of a 1 link between any pair of groups

P.S. Koutsourelakis, Non-Identifiability A B 100% group 1 50% group 1 50% group 2  Different configurations (with 2, 3 or 4 groups) have the same likelihood  Prior determines inference results

P.S. Koutsourelakis, Application: Mixed-Membership 1 domains – 16 objects 4 distinct identities fully observed adjacency matrix

P.S. Koutsourelakis, Application: Mixed-Membership Model

P.S. Koutsourelakis, Application: Mixed-Membership Model

P.S. Koutsourelakis, Application: Mixed-Membership Model

P.S. Koutsourelakis, Application: Mixed-Membership IRM MMM Error w.r.t. actual probability that any pair of objects belong to the same group

P.S. Koutsourelakis, Application: Mixed-Membership IRM MMM

P.S. Koutsourelakis, Application: Mixed-Membership Model 2 domains (animals + features) Animals form two groups: birds + 4-legged mammals

P.S. Koutsourelakis, Application: Mixed-Membership Model

P.S. Koutsourelakis, Application: Mixed-Membership Model COW: Average posterior pairwise probabilities of belonging to the same group

P.S. Koutsourelakis, 34 people A disagreement between administrator (34) and instructor (1) led to the split of the club in two (circles and squares) Used binary matrix that records “like” relation Zachary’s Karate Club from M Girvan and MEJ Newman, Proc. Natl. Acad. Sci. USA, 2002

P.S. Koutsourelakis, Zachary’s Karate Club

P.S. Koutsourelakis, Learning Hierarchies Can we meaningfully infer a hierarchy of groups/identities? Identity 1 Identity 2 Identity 3 Identity 4 most general most specific

P.S. Koutsourelakis, Learning Hierarchies Nonparametric prior on trees Level 0 Level L Level L-1 each box is a different group/identity CRP L (a L ) CRP L-1 (a L-1 )

P.S. Koutsourelakis, Learning Hierarchies “Forward” interpretation (for a single domain) Hierarchical Mixed Membership Model (HMMM)

P.S. Koutsourelakis, Application: Artificial Dataset 1 domain – 40 objects 4 distinct identities fully observed adjacency matrix

P.S. Koutsourelakis, Application: Artificial Dataset

P.S. Koutsourelakis, Application: Political Books  43 liberal, 49 conservative, 13 neutral  Links imply frequent co-purchasing by the same buyers (Amazon.com)

P.S. Koutsourelakis, Application: Political Books 27% 50% 23% 19% 46% 35% 0% 100% 8% 84% 6% 94% 0% 100% 0% 100% 0% 100% 0% 100%

P.S. Koutsourelakis, Reality Mining MIT Data  1 node type (people)  97 people + all outsiders in one node  22 different positions (professor,staff,1styeargrad,….) sloan 29% faculty& staff 5% students 52% other 14%

P.S. Koutsourelakis, Reality Mining MIT Data 12% 64% 12% 50% 33% 17 % 0%0% 7% 86% 7% 0% 4% 0% 96% 100% 0% 100% 0% 33% 0% 50 % 17 % 15% 83% 4%4% 8%8% 25% 50% 0% 25%

P.S. Koutsourelakis, Conclusions and Outlook Relational data contain significant information about group structure Bayesian models allow the analyst to make inferences about communities of interest while quantifying the level of confidence, even when a significant proportion of the data is missing Nonparametric models provide a way of getting very flexible priors that allow the model to adapt to the data. IRM is a very lightweight framework with a very wide range of applicability, but cannot capture multiple identities. MMM and HMMM allows for increased flexibility and provides additional information about objects that simultaneously belong to several groups. Challenges:  Accelerated inference especially when dealing with large datasets: - Variational methods - Sequential Monte Carlo  Appropriate priors for time dependent datasets are needed

P.S. Koutsourelakis, Application: Senate Vote 2002  50 Democrats, 49 Republicans, 1 Independent  Link R i,j =1 if: - voted the same - have both taken more or less than the average contribution average: $13,800

P.S. Koutsourelakis, Application: Senate Vote % 100% 0% 67% 33% 0% 25% 75% 3% 12% 85% 0% 70% 30%