1. Outline 1.Introduction 2.Harvesting Classes 3.Harvesting Facts 4.Common Sense Knowledge 5.Knowledge Consolidation 6.Web Content Analytics 7.Wrap-Up from the crowd from the Web from images from KBs 1

2. Goal: Commonsense Knowledge Apples are green, red, round, juicy, … but not fast, funny, verbose, … Pots and pans are in the kitchen or cupboard, on the stove, … but not in in the bedroom, in your pocket, in the sky, … People have only one spouse (at a time) Our goal is to find common sense knowledge such as Problem: This information is rarely mentioned explicitly on the Web "blind person": 570,000 results on Google "non-blind person": 3,300 results on Google "green cherry": 245m results "red cherry": 70m results "red cow": 31.9m results "brown cow": 31.8m results "green cow": 31.7m results 2

3. Verbosity Game [Speer & Havasi 2012] http://www.gwap.com/gwap/ 3

4. Concept Net many inputs incl. WordNet, Verbosity game, etc. ConceptNet 5: 3.9 Mio concepts 12.5 Mio. edges oven cook restauran t satisfy hunge r follow recipe bake cake person survive eat desert sweet isA usedFor atLocation usedFor causes- Desire createdBy capableOf desires motivatedByGoal desires, capableOf atLocation usedFor isA hasProperty recievesAction motivatedByGoal [Speer & Havasi 2012] http://conceptnet5.media.mit.edu/ 4

5. Outline 1.Introduction 2.Harvesting Classes 3.Harvesting Facts 4.Common Sense Knowledge 5.Knowledge Consolidation 6.Web Content Analytics 7.Wrap-Up from the crowd √ from the Web from images from KBs 5

6. Pattern-Based Harvesting of Commonsense Properties Start with seed facts for apple hasProperty round dog hasAbility bark plate hasLocation table Find patterns that express these relations, such as X is very Y, X can Y, X put in/on Y, … Problem: noise and sparseness of data Solution: harness Web-scale n-gram corpora  5-grams + frequencies Confidence score: PMI (X,Y), PMI (p,(XY)), support(X,Y), … are features for regression model (N. Tandon et al.: AAAI 2011) Apply these patterns to find more facts. 6

7. Commonsense Properties with Semantic Types (N. Tandon et al.: WSDM 2014) 1) Compute the ranges for common-sense relations hasTaste: sweet, sour, spicy, … 2) Compute the domains for common-sense relations hasTaste: shake (milk shake), juice… 3) Compute assertions hasTaste: { shake/sweet, … } For all 3 tasks, use label propagation on a graph with few seeds from WordNet and with edges from n-gram corpus.  WebChild: 4 Mio. triples for 19 relations www.mpi-inf.mpg.de/yago-naga/webchild 7 univsch>

8. Universal Schema Rome,Italy Paris,France Florence,Italy … is the captial of is located in 's location in capitalOf locatedIn [Riedel et al.: HLT-NAACL’13] patterns in text relations in KB # co-occurrences sub-relations equivalent relations symmetry facts sub-patterns equivalent patterns Do matrix completion, learn: 8

9. Outline 1.Introduction 2.Harvesting Classes 3.Harvesting Facts 4.Common Sense Knowledge 5.Knowledge Consolidation 6.Web Content Analytics 7.Wrap-Up from the crowd √ from the Web √ from images from KBs 9 img>

10. ImageNet Populate WordNet classes with photos [J. Deng et al.: CVPR‘09] How: crowdsourcing for seeds, distantly supervised classifiers, object recognition (bounding boxes) in computer vision http://www.image-net.org 10

11. NEIL Infer instances of partOf occursAt, inScene relations [X. Chen et al.: ICCV‘13] http://www.neil-kb.com/ bike inScene trafficJam bike occursAt park pedals partOf bike 11

12. Outline 1.Introduction 2.Harvesting Classes 3.Harvesting Facts 4.Common Sense Knowledge 5.Knowledge Consolidation 6.Web Content Analytics 7.Wrap-Up from the crowd √ from the Web √ from images √ from KBs 12

13. Path Ranking Algorithm (PRA) isA- playsInLeague ABC Football player isA Batistuta hasTeam isInLeague (Lao et al.: EMNLP’11) plays- inLeague Observation: There are often alternative ways to walk from the subject of a fact to its object: hasTeam + isInLeague isA + isA- + playsInLeague playsInLeague(Batistuta, A) 13

14. Path Ranking Algorithm (PRA) isA- playsInLeague ABC Football player isA Batistuta hasTeam isInLeague (Lao et al.: EMNLP’11) plays- inLeague Use regularized logistic regression to learn a weight θ P indicating how often a path leads to the correct object: hasTeam + isInLeague60% isA + isA- + playsInLeague10% playsInLeague(Batistuta, A) 14

15. Path Ranking Algorithm (PRA) isA- playsInLeague ABC Football player isA Caruzzo hasTeam isInLeague (Lao et al.: EMNLP’11) playsInLeague(Caruzzo, A) ? path P Score(s,t)= Σ f P (s,t)θ P f P (s,t)=Pr(s → t ; P) 15

16. Learn Horn Rules Problem: Rule mining needs counter-examples, but RDF KBs are positive only. Should not use absent information as counter-examples because of the Open World Assumption. 16

17. Partial Completeness Assumption Partial Completeness Assumption: If we know r(x,y1),..., r(x,yn), then all other r(x,z) are false (used as "Local closed world assumption" in Google KV) [Galarraga et al.: WWW’13] 17

18. AMIE: Efficient Rule Mining [Galarraga et al.: WWW’13] AMIE (efficient search space traversal with pruning) 18 relwork>

19. More Related Work Sensor factorization (Harshman et al., '94) Probabilistic Relational Learning (Friedman et al., '99) Relational Markov Networks (Taskar et al., '02) Markov-logic Networks (Kok et al., '07) Extension of SBMs (Kemp et al., '06) (Sutskever et al., '10) Spectral clustering (undirected graphs) (Dong et al., '12) Collective matrix factorization (Nickel et al., '11) Embedding models (Bordes et al., '11, '13) (Jenatton et al., '12) (Socher et al., '13) (Wang et al., '14) (Garca-Duran et al., '14) (Weston et al., '14) Jason Weston’s slide @ AKBC’14 19

20. Open Problems and Grand Challenges 20 Commonsense rules beyond Horn clauses Comprehensive commonsense knowledge organized in ontologically clean manner especially for emotions and other analytics Visual knowledge with text grounding: populate concepts, typical activities & scenes could serve as training data for image & video understanding  x: type(x,spider)  numLegs(x)=8  x: type(x,animal)  hasLegs(x)  even(numLegs(x))  x: human(x)  (  y: mother(x,y)   z: father(x,z))  x: human(x)  (male(x)  female(x))