1. Infinite block models for belief networks, social networks, and cultural knowledge Josh Tenenbaum, MIT 2007 MURI Review Meeting Work of Charles Kemp, Chris Baker, Tom Griffiths, Pat Shafto, Vikash Mansinghka, Dan Roy

2. Goal Algorithmic tools for uncovering structure in belief networks, social networks, and joint structure (social-belief networks). Why? –Joint social-belief structure ~ culture –Algorithms let us map cultural knowledge quickly and semi-automatically, detect changes and track dynamics.

3. Approach Data –People’s beliefs about properties of objects –Relations between people –People’s beliefs about relations between objects (or people). Representation: cluster-based models –Clusters of things: categories –Clusters of people: social groups –Clusters of people who share similar beliefs about clusters of things (or people): cultural groups

4. Approach Learning: Bayesian inference from data –Relational models: analyze arbitrary relational databases of beliefs, not just a single matrix –Nonparametric models: automatically determine complexity of representations –Hierarchical models: multiple levels of structure –Nested models: structures with structure Result: a flexible toolkit that goes qualitatively beyond standard algorithms. –e.g., ability to discover cultural groups characterized by a shared understanding of the environment’s relational structure.

5. Talk outline Classic cluster-based methods New methods –Clustering with arbitrary relational systems –Hierarchical relational clustering –Cross-cutting clustering with nested models –Cross-cutting relational clustering Application to Guatemala data from Atran & Medin Conclusions and future directions

6. Classic cluster-based methods Belief networks: mixture models

7. Classic cluster-based methods Belief networks: mixture models

8. Classic cluster-based methods Social networks: block models DefersTo(P i, P j )

9. Classic cluster-based methods Cultural knowledge (joint social/belief structure): cultural consensus model Not cluster- based. SVD on matrix of people x questions

10. Problems with classic methods No principled tools for discovering different cultural groups characterized by different belief networks. –CCM not intended to find cultural groups, but rather to uncover (and test for) shared knowledge and authoritativeness in a single cultural group. “Test theory without an answer key” Can only represent simple forms of knowledge that fit into a single two-mode matrix. –Cultural knowledge is often much richer….

11. Talk outline Classic cluster-based methods New methods –Clustering with arbitrary relational systems –Hierarchical relational clustering –Cross-cutting clustering with nested models –Cross-cutting relational clustering Application to Guatemala data from Atran & Medin Conclusions and future directions

12. people social relation Alyawarra tribe, central Australia (Denham) –104 individuals –27 binary social relations –3 attributes: kinship class, age, sex (used only for cluster validation, not learning) people attributes Clustering arbitrary relational systems

13. Infinite relational model (IRM) discovers 15 clusters Clustering arbitrary relational systems

14. International relations circa 1965 (Rummel) –14 countries: UK, USA, USSR, China, …. –54 binary relations representing interactions between countries: exports to( USA, UK ), protests( USA, USSR ), …. –90 (dynamic) country features: purges, protests, unemployment, communists, # languages, assassinations, ….

16. concept predicate Data from UMLS (McCrae et al.) medical knowledge base: –134 terms: enzyme, hormone, organ, disease, cell function... –49 predicates: affects(hormone, organ), complicates(enzyme, cell function), treats(drug, disease), diagnoses(procedure, disease) … Clustering arbitrary relational systems

17. Conceptual clusters discovered: Clustering arbitrary relational systems Relations between clusters:

18. Hierarchical relational clustering

19. Learnig a hierarchical ontology (Roy, Kemp, Mansinghka & Tenenbaum)

20. Models so far all learn a single system of clusters. We would like to be able to discover multiple cross-cutting systems of clusters. –Within an individual’s mind: multiple mental models of a single complex domain. –Across individuals in a population: multiple cultural groups with different characteristic mental models. Cross-cutting clustering with nested models

21. Conventional mixture model Cross-cutting clustering with nested models

22. CrossCat model Cross-cutting clustering with nested models

23. Analysis of US Senate votes 1989-90 101 senators x 638 issues 10 systems of classes. Core democratic platform “Hot-button” social issues Law and orderMilitary Environment & agriculture

24. Nested relational model: Cross-cutting clustering with nested models people relation Infinite relational model: people relation

25. Discovering cultural groups based on shared relational knowledge Guatemala studies of Atran & Medin –Subjects 12 native Itza’ maya 12 immigrant Ladino 12 immigrant Q’eqchi’ maya –Questions Does plant i help animal j? animal plant people Nested relational model:

26. Discovering cultural groups based on shared relational knowledge I1 I2 I3 I5 I7 I8 I9 I10 I12 L1 L2 L3 L4 L5 L6 L7 L8 L9 L10 L11 L12 I6 I11 Clusters of people found: Guatemala studies of Atran & Medin –Subjects 12 native Itza’ maya 12 immigrant Ladino 12 immigrant Q’eqchi’ maya –Questions Does plant i help animal j? Q3 Q6 Q8 Q9 Q10 Q11 Q12 Q1 Q2 Q4 Q5 Q7 I4

27. Talk outline Classic cluster-based methods New methods –Clustering with arbitrary relational systems –Hierarchical relational clustering –Cross-cutting clustering with nested models –Cross-cutting relational clustering Application to Guatemala data from Atran & Medin Conclusions and future directions

28. Conclusions A flexible toolkit for statistical learning about cultural knowledge and cultural groups. –Relational models: analyze arbitrary relational databases of beliefs, not just a single matrix –Nonparametric models: automatically determine complexity of representations –Hierarchical models: multiple levels of structure –Nested models: structures with structure Can automatically discover important qualitative structure in real-world data (Atran & Medin, DARPA CPoF).

29. Ongoing and future work Algorithms that can scale to very large networks. More dynamic data and models. –Second-generation Guatemala data –Political data sets: voting records, international relations Better statistical models for sparse networks. More structured representations necessary to capture “cultural stories”: grammars, logical schemas. Multi-level statistical models for learning about network structure from raw event data.

30. Learning network structure from raw event data edge (N) class (Z) edge (N) 123456 78910111213141516 # of samples: 20 80 1000 Data D Network N Data D Network N Abstract Classes 1 2 3 4 5 6 … 7 8 9 10 11 12 13 14 15 16 … … 0.4 0.0 … … (Mansinghka, Kemp, Tenenbaum, Griffiths UAI 06) c1c1 c2c2 c1c1 c2c2 c1c1 c2c2 Classes Z 

31. edge (N) class (Z) edge (N) 1 2 3 4 5 6 7 8 9 10 11 12 # of samples: 40 100 1000 Data D Network N Data D Network N Abstract Classes 1 2 3 4 5 6 7 8 9 10 11 12 … 0.1 c1c1 c1c1 c1c1  Classes Z … … … Learning network structure from raw event data (Mansinghka, Kemp, Tenenbaum, Griffiths UAI 06)

32. Learning abstract structure in networks Primate troop Bush administration Prison inmates New Guinea islands “beats” “told” “likes” “trades with” Dominance hierarchy Tree Cliques Ring

33. The end

35. Discovering structure in relational data 3 9 1 13 5 11 7 14 2 10 6 12 4 8 15 3 9 1 13 5 11 7 14 2 10 6 12 4 8 15 3 9 1 13 5 11 7 14 2 10 6 12 4 8 15 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 InputOutput person TalksTo(person,person) person

36. O z Infinite Relational Model (IRM)  3 9 1 13 5 11 7 14 2 10 6 12 4 8 15 0.9 0.1 0.9 0.1 3 9 1 13 5 11 7 14 2 10 6 12 4 8 15 3 9 1 13 5 11 7 14 2 10 6 12 4 8 15

37. Model fitting

38. a(3). a(9). a(1). a(13). a(5). a(11). b(7). b(14). b(2). b(10). b(6). c(12). c(4). c(8). c(15). r(X,Y) ← a(X),a(Y). (0.0) r(X,Y) ← a(X),b(Y). (0.9) r(X,Y) ← c(X),a(Y). (0.95)... r(3,7). r(1,10). r(2,4)... The concepts discovered by the IRM can serve as primitives in complex logical theories (cf. clustering approaches to predicate invention, e.g., Craven and Slattery (2001) or Popescul and Ungar (2004)) 3 9 1 13 5 11 7 14 2 10 6 12 4 8 15 3 9 1 13 5 11 7 14 2 10 6 12 4 8 15

39. Related Work Relational models –Sociology: Wang and Wong (1987); Nowicki and Snijders (2001) –Machine learning: Taskar, Segal and Koller (2001) Wolfe and Jensen (2004) Wang, Mohanty and McCallum (2005) Nonparametric Bayesian models Ferguson (1973); Neal (1991) Nonparametric Bayesian relational models Carbonetto, Kisynski, de Freitas and Poole (2005) Xu, Tresp, Yu, Kriegel (2006)

40. Optimization (or inference) Global proposals –Split and merge clusters Local proposals –Re-assign one entity to best cluster based on current assignments of all other entities (i.e., Gibbs sampling) Both cognitively plausible and computationally reasonable.

42. O z Infinite relational model (IRM)  3 9 1 13 5 11 7 14 2 10 6 12 4 8 15 0.9 0.1 0.9 0.1 3 9 1 13 5 11 7 14 2 10 6 12 4 8 15 3 9 1 13 5 11 7 14 2 10 6 12 4 8 15

43. O z Infinite relational model (IRM)  3 9 1 13 5 11 7 14 2 10 6 12 4 8 15 0.9 0.1 0.9 0.1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

44. Independent symmetric beta priors on  : Chinese Restaurant Process over z: Goal: –Infer –Infer (integrating out  to reduce space of unknowns) Generating  and z

45. Global-local search process

46. Joint modeling of belief systems and social systems animal plant person helps(plant,animal,person judging) Data from Atran and Medin

48. ItzaLadinos