A Review of Relational Machine Learning for Knowledge Graphs CVML Reading Group Xiao Lin
M. Nickel, K. Murphy, V. Tresp, E. Gabrilovich. “A Review of Relational Machine Learning for Knowledge Graphs. ” arXiv
VQA Knowledge base MRF vs embeddings 3
The relational learning problem Neural embedding models 4
“Leonard Nimoy was an actor who played the character Spock in the science-fiction movie Star Trek” 5
subjectpredicateobject Leonard NimoyprofessionActor Leonard Nimoystarred inStar Trek Leonard NimoyplayedSpock character inStar Trek genreScience-fiction “SPO triples” 6
“Leonard Nimoy was an actor who played the character Spock in the science-fiction movie Star Trek” Leonard Nimoy Star Trek Science-fiction profession starred in played character in genre Leonard Nimoy Spock Star Trek Actor Spock 7 Knowledge Graphs
“Leonard Nimoy was an actor who played the character Spock in the science-fiction movie Star Trek” Spock Leonard Nimoy Actor Star Trek Science-fiction profession starred in played character in genre 8
Knowledge Graphs a.k.a. Relational data a.k.a. Linked data Spock Leonard Nimoy Actor Star Trek Science-fiction profession starred in played character in genre 9
Knowledge Graphs a.k.a. Semantic network a.k.a. Knowledge base a.k.a. Semantic web a.k.a. Big data a.k.a. Knowledge graph Spock Leonard Nimoy Actor Star Trek Science-fiction profession starred in played character in genre (1960s) (1970s) (2000s) (2010s) (2012) a.k.a. Artificial Intelligence 10
Query Knowledge Graphs Spock Leonard Nimoy Actor Star Trek Science-fiction profession starred in played character in genre Was Leonard Nimoy in Star Trek? Leonard Nimoy starred in Star Trek 11
Query Knowledge Graphs Spock Leonard Nimoy Actor Star Trek Science-fiction profession starred in played character in genre Was Leonard Nimoy in Star Trek? Leonard Nimoy starred in Star Trek True 12
Query Knowledge Graphs Spock Leonard Nimoy Actor Star Trek Science-fiction profession starred in played character in genre Which character did Leonard Nimoy play in Star Trek? Leonard Nimoy played (_X_) (_X_) character in Star Trek 13
Query Knowledge Graphs Spock Leonard Nimoy Actor Star Trek Science-fiction profession starred in played character in genre Which character did Leonard Nimoy play in Star Trek? Leonard Nimoy played (Spock) (Spock) character in Star Trek True 14
Query Knowledge Graphs Spock Leonard Nimoy Actor Star Trek Science-fiction profession starred in played character in genre Which character did Leonard Nimoy play in Star Trek? Leonard Nimoy played (Spock) (Spock) character in Star Trek True 15
Query Knowledge Graphs Spock Leonard Nimoy Actor Star Trek Science-fiction profession played character in genre Was Leonard Nimoy in Star Trek? Leonard Nimoy starred in Star Trek 16
Relational Machine Learning Goal: Infer relationships between objects and answer queries Database of (partially) annotated relations 17
Relational Machine Learning Goal: Infer relationships between objects and answer queries Database of (partially) annotated relations Learn rules or models * * 18
Relational Machine Learning Goal: Infer relationships between objects and answer queries Database of (partially) annotated relations Answer queries on unseen relations Learn rules or models 19 * *
Relational Machine Learning Goal: Infer relationships between objects and answer queries Database of (partially) annotated relations Answer queries on unseen relations Learn rules or models WordNet Wikipedia Freebase YAGO NELL ReVerb Knowledge Vault Web …. ILP MLN Random walk Word2Vec Tensor factorization Neural embeddings … 20
Relational Machine Learning Applications Bioinformatics Given that: Some DNA produces some protein Some protein has some function Some protein interacts some other protein What does this protein do? 21
Relational Machine Learning Applications Bioinformatics Question Answering (QA) “Find a table for 4 tonight in Chicago.” “How’s the weather tomorrow?” “Read my latest s” “Why did the chickens cross the road?” 22
Relational Machine Learning Applications Bioinformatics Question Answering (QA) Search 23
Relational Machine Learning Applications Bioinformatics Question Answering (QA) Search 24
Relational Machine Learning Applications Bioinformatics Question Answering (QA) Search Visualization [Wang et al. 2012] Reviews for La Baguette Bakery, Stanford, CA 25
Relational Machine Learning Applications Bioinformatics Question Answering (QA) Search Visualization Social networks Recommendation Controversial stuff program-may-be-killing-thousands-of-innocent-people/ 26
Relational Machine Learning What I really do 27
Relational Machine Learning Goal: Infer relationships between objects and answer queries Database of (partially) annotated relations Answer queries on unseen relations Learn rules or models 28 * *
Relational Machine Learning Goal: Infer relationships between objects and answer queries Database of (partially) annotated relations Answer queries on unseen relations Learn rules or models WordNet Wikipedia Freebase YAGO NELL ReVerb Knowledge Vault Web …. ILP MLN Random walk Word2Vec Tensor factorization Neural embeddings … 29
Relational Machine Learning Challenges Database of (partially) annotated relations Quantity Collaborative Automatic IE Negative examples Locally closed world Noise Grounding Barrack Obama /m/03gh4 Bias 30
Relational Machine Learning Challenges Database of (partially) annotated relations Learn rules or models Quantity Collaborative Automatic IE Negative examples Locally closed world Noise Grounding Barrack Obama /m/03gh4 Bias Accuracy Efficiency Scalability Interpretability 31
Relational Machine Learning Challenges Database of (partially) annotated relations Answer queries on unseen relations Learn rules or models Quantity Collaborative Automatic IE Negative examples Locally closed world Noise Grounding Barrack Obama /m/03gh4 Bias Accuracy Efficiency Scalability Interpretability Fast inference Scale to complex queries Conjunctions 32
Relational Machine Learning Models Symbolic vs. Connectionist 33
Relational Machine Learning Models Symbolic vs. Connectionist MRF vs. Embedding MRF (Markov Logic Nets) Spock Leonard Nimoy Actor Star Trek Science-fiction profession played character in genre Unobserved relations are jointly inferred Learn a set of rules (factors) and weights (MCMC) Infer all unobserved relations jointly by maximizing probability This is not an MRF
Relational Machine Learning Models Symbolic vs. Connectionist MRF vs. Embedding MRF Embedding (RESCAL) Spock Leonard Nimoy Actor Star Trek Science-fiction profession played character in genre Unobserved relations are independent given SPO embeddings Learn SPO embeddings (matrix/tensor factorization, neural nets) Predict relations SPO Embeddings 35
1 Cat Dog 1 Cat Dog 1 36
Relational Machine Learning Models Symbolic vs. Connectionist MRF vs. Embedding (S,P,O)=0/1 classification performance [Nickel et al. 2015] 37
Relational Machine Learning Models Symbolic vs. Connectionist MRF vs. Embedding Graph mining Predict relations using graph features, usually independently given graph features In degree, out degree Common ancestor Random walk S->O paths … Spock Leonard Nimoy Actor Star Trek Science-fiction profession played character in genre 38
Neural Embedding Models Predicting a score based on Subject, Object, Predicate embeddings 39
Neural Embedding Models Predicting a score based on Subject, Object, Predicate embeddings S O P LookupTable 40
Neural Embedding Models Predicting a score based on Subject, Object, Predicate embeddings Something that allows backprop S O P LookupTable 41
Neural Embedding Models Predicting a score based on Subject, Object, Predicate embeddings Something that allows backprop S O LookupTable P 42
Neural Embedding Models Predicting a score based on Subject, Object, Predicate embeddings ER-MLP [Dong et al. 2014] Linear E-MLP Linear 43
Neural Embedding Models Predicting a score based on Subject, Object, Predicate embeddings Linear -L1 distance Linear Structured Embedding (SE) [Bordes et al. 2011] CAddTable -L2 distance TransE [Bordes et al. 2013] CAddTable -L2 distance TransR [Lin et al. 2015] Linear 44
Neural Embedding Models Predicting a score based on Subject, Object, Predicate embeddings RESCAL [Nickel et al. 2011] NTN [Socher et al. 2013] Linear HolE [Nickel et al. 2015] 45
Neural Embedding Models WordNet (WN18) Words S&O (words) 18 Predicates (relations) triples Search Engine GoogleYahooAsk Jeeves Has_instance Trademark Direct hypernym 46
Neural Embedding Models WordNet (WN18) Words S&O (words) 18 Predicates (relations) triples Freebase (FB15k) General facts S&O (words) 1345 Predicates (relations) triples Top_equivalent_webpage Composition type Piano sonata Pathétique Also_known_as Opus 13 Notable_for /type/object/key Pianosonates 47
Neural Embedding Models WordNet (WN18) Words S&O (words) 18 Predicates (relations) triples Freebase (FB15k) General facts S&O (words) 1345 Predicates (relations) triples (?,P,O) => S from all possible {S} 48
Neural Embedding Models WordNet (WN18) Words S&O (words) 18 Predicates (relations) triples Freebase (FB15k) General facts S&O (words) 1345 Predicates (relations) triples (?,P,O) => S from all possible {S} Locally Closed World Assumption (LCWA) Multiple S Raw Filter: remove other true S’ 49
Neural Embedding Models WordNet (WN18) Words S&O (words) 18 Predicates (relations) triples Freebase (FB15k) General facts S&O (words) 1345 Predicates (relations) triples Mean Reciprocal Rank (MRR) 50
Neural Embedding Models [Nickel et al. 2015] WordNet (filtered?) MethodFilteredRaw1310 ER-MLP TransE TransR RESCAL HolE
Neural Embedding Models [Nickel et al. 2015] Freebase (filtered?) MethodFilteredRaw1310 ER-MLP TransE TransR RESCAL HolE
Neural Embedding Models [Nickel et al. 2015] Freebase (filtered?) MethodFilteredRaw1310 ER-MLP TransE TransR RESCAL HolE
Neural Embedding Models Qualitative: Word2Vec visualizations (again) [Mikolov et al. 2013] 54
Big Knowledge Graphs Confident predictions [Dong et al. 2014] Prob(SPO)>0.9 55
Big Knowledge Graphs Confident predictions Google Knowledge Vault [Dong et al. 2014] Web information extraction Neural embedding Path ranking 56
Knowledge base and VQA Incorporating entity and relation embeddings Improving VQA model When entities and relations become compositional 57