1. C O R P O R A T E T E C H N O L O G Y Information & Communications Intelligent Autonomous Systems Infinite Hidden Relational Models Zhao Xu 1, Volker Tresp 2, Kai Yu 2, Shipeng Yu and Hans-Peter Kriegel 1 1 University of Munich, Germany 2 Siemens Corporate Technology, Munich, Germany

2. © Siemens AG, CT IC 4 Motivation Relational learning is an object oriented approach to representation and learning that clearly distinguishes between entities (e.g., objects), relationships and their respective attributes and represents an area of growing interest in machine learning Learned dependencies encode probabilistic constraints in the relational domain Many relational learning approaches involve extensive structural learning, which is makes RL somewhat tricky to apply in practice The goal of this work is an easy to apply generic system which relaxes the need for extensive structural learning In the infinite hidden relational model (IHRM) we introduce for each entity an infinite-dimensional latent variable, whose state is determined by a Dirichlet process The resulting representation is a network of interacting DPs

3. © Siemens AG, CT IC 4 Work on DPs in Relational Learning C. Kemp, T. Griffiths, and J. R. Tenenbaum (2004). Discovering Latent Classes in Relational Data (Technical Report AI Memo 2004-019) Kemp, C., Tenenbaum, J. B., Griffiths, T. L., Yamada, T. & Ueda, N. (2006). Learning systems of concepts with an infinite relational model. AAAI 2006 Z. Xu, V. Tresp, K. Yu, S. Yu, and H.-P. Kriegel (2005). Dirichlet enhanced relational learning. In Proc. 22 nd ICML, 1004-1011. ACM Press Z. Xu, V. Tresp, K. Yu, and H.-P. Kriegel. Infinite hidden relational models. In Proc. 22nd UAI, 2006 P. Carbonetto, J. Kisynski, N. de Freitas, and D. Poole. Nonparametric bayesian logic. In Proc. 21st UAI, 2005.

4. © Siemens AG, CT IC 4 Ground Network With an Image Structure Ground Network A: entity attributes R: relational attributes (e.g., exist, not exist) Limitations Attributes locally predict the probability of a relational attribute Given the parent attributes, all relational attributes are independent To obtain non local dependency: structural learning might be involves

5. © Siemens AG, CT IC 4 Ground Network With an Image Structure and Latent Variables: The HRM Z: latent variable Information can now flow through the network of latent variables In an IHRM, Z can be thought of as representing unknown attributes (such as a cluster attribute) Note, that in image processing, Z would correspond to the true pixel value, A to a noisy measurement and R would encode neighboring pixel value constraints

6. © Siemens AG, CT IC 4 A Recommendation System items users A relational attribute (like) only depends on the attributes of the user and the item If both attributes are weak, we’re stuck A relational attribute (like) only depends on the states of the latent variables of user and item If entity attributes are weak, other known relations are exploits, we exploit collaborative information items users

7. © Siemens AG, CT IC 4 The Hidden Relational Model (HRM) items users A relational attribute (like) only depends on the states of the latent variables of user and item If entity attributes are weak, other known relations are exploits, we exploit collaborative information G0bG0b G0uG0u G0mG0m Multinomial with Dirichlet priors; Three Base Distributions For a DP model, the number of states becomes infinite; the prior distribution for is denoted as The Infinite Hidden Relational Model (IHRM)

8. © Siemens AG, CT IC 4 Inference in the IHRM 1.Gibbs sampler derived from the Chinese restaurant process representation (Kemp et al. 2004, 2006, Xu et al. 2006); 2.Gibbs sampler derived finite approximations to the stick breaking representation 1.Dirichlet multinomial allocation 2.Truncated Dirichlet process 3.Two mean field approximations based on those procedures 4.A memory-based empirical approximation (EA) (2,3,4 in Xu et al 2006, submitted)

9. © Siemens AG, CT IC 4 Generative Model with CRP

10. © Siemens AG, CT IC 4 Generative Model with Truncated DP

11. © Siemens AG, CT IC 4 Inference (1): Gibbs Sampling with CRP

12. © Siemens AG, CT IC 4 Inference (2): Gibbs Sampling with Truncated DP

13. © Siemens AG, CT IC 4 Inference (3): Mean Field with Truncated DP

14. © Siemens AG, CT IC 4 Experimental Analysis on Movie Recommendation (1) Task description To predict whether a user likes a movie given attributes of users and movies, as well as known ratings of users. Data set: MovieLens Model User Like User Attributes Movie R Attributes ZuZu ZmZm G0uG0u G0mG0m G0bG0b

15. © Siemens AG, CT IC 4 Experimental Analysis on Movie Recommendation (2) Result Method Prediction Accuracy (%) Time (s)#Comp u #Comp m given5given10given15given20 GS-TDP65.7166.4766.9968.33234976741 MF-TDP65.0665.3866.5467.69101496 EA63.9164.1064.55 386--- Note, for GS-TDP and MF-TDP, α 0 =100 943 users, 1680 movies

16. © Siemens AG, CT IC 4 Experimental Analysis on Gene Function Prediction (1) Task description To predict functions of genes given the information on the gene- level and the protein-level, as well as interaction between genes. Data set: KDD Cup 2001 Model

17. © Siemens AG, CT IC 4 Gene Interact ZgZg R g,g Gene Attributes Phenotype ZpZp Observe R g,p Structural Category Z cl belong R g,cl Motif ZmZm Contain R g,m Complex ZcZc Form R g,c Function ZfZf Have R g,f Experimental Analysis on Gene Function Prediction (2)

18. © Siemens AG, CT IC 4 An example gene AttributeValue Gene IDG234070 EssentialNon-Essential Structural Category1, ATPases 2, Motorproteins ComplexCytoskeleton PhenotypeMating and sporulation defects MotifPS00017 Chromosome1 Function 1, Cell growth, cell division and DNA synthesis 2, Cellular organization 3, Cellular transport and transport mechanisms Experimental Analysis on Gene Function Prediction (3)

19. © Siemens AG, CT IC 4 Results AlgorithmAccuracy(%)#Comp gene GS-TDP89.4615 MF-TDP91.96742 EA93.18--- Kdd cup winner 93.63--- Experimental Analysis on Gene Function Prediction (4)

20. © Siemens AG, CT IC 4 Results Relationships Prediction Accuracy (%) (without the relationship) Importance Complex91.13197 Interaction92.14100 Structural Category92.6155 Phenotype92.7145 Attributes of Gene93.0810 Motif93.126 The importance of a variety of relationships in function prediction of genes Experimental Analysis on Gene Function Prediction (5)

21. © Siemens AG, CT IC 4 Experimental Analysis on Clinical Data (1) Task description To predict future procedures for patients given attributes of patients and procedures, as well as prescribed procedures and diagnosis of patients. Model

22. © Siemens AG, CT IC 4 Patient Take Patient Attributes Procedure R pa,pr  pa,pr Procedure Attributes Z pa Z pr  pa  pa θ pr  pr α 0 pa G 0 pa G 0 pr G 0 pa,pr α 0 pr Make Diagnosis R pa,dg  pa,dg Diagnosis Attributes Z dg θ dg  dg G 0 dg G 0 pa,dg α 0 dg Experimental Analysis on Clinical Data (2)

23. © Siemens AG, CT IC 4 Experimental Analysis on Clinical Data (3) Results ROC curves for predicting procedures, average on all patients ROC curves for predicting procedures, only considering patients with prime complaint circulatory problem E1: one-sided CF E2: 2-sided CF E3: full model E4: no hidden E5: content based BN

24. © Siemens AG, CT IC 4 Conclusion The IHRM is a new nonparametric hierarchical Bayes model for relational modeling Advantages Reducing the need for extensive structural learning Expressive ability via coupling between heterogeneous relationships The model decides itself about the optimal number of states for the latent variables. Scaling: # of entities times # of occupied states times # of known relations Note: default relations (example: by default there is no relation) can often be treated as unknown and drop out Conjugacy can be exploited

25. © Siemens AG, CT IC 4 A memory-based empirical approximation First, we assume the number of components to be equal to the corresponding entities in the corresponding entity class Then in the training phase each entity contributes to its own class only Based on this simplification the parameters in the attributes and relations can be learned very efficiently. Note that this approximation can be interpreted as relational memory-based learning To predict a relational attributes we assume that only the states of the latent variables involved in the relation are unknown