1. 1 Extracting Semantic Networks From Text Via Relational Clustering Stanley Kok Dept. of Computer Science and Eng. University of Washington Seattle, USA Joint work with Pedro Domingos

2. Goal: Reading & Understanding Text Semantic Network country invade people emigrate to embrace religion Semantic Network Extractor A Step Towards the Goal TextRunner [Banko et al., IJCAI’07] (Object,Relation,Object) triples Webpages 2 Text Autonomous Agent Knowledge Base man(x)=>human(x) child(x,y)=>parent(y,x) …

3. 33 Snippet of Extracted Semantic Network America US USA Australia Britain UK China Spain Iraq Germany … EU European_Union UN United_Nations army troops force forces navy export import imports importation part role pulled withdrew to_withdraw to_remove banned had_banned prohibited restricted banned had_banned prohibited restricted played has_played will_play … played has_played will_play …

4. 4 Motivation Supervised approaches Manual annotation of training data; not scalable to Web e.g., Semantic Parsing [Wong & Mooney, ACL’07] Unsupervised approaches Extracts noisy & sparse ground facts; no high-level knowledge that generalizes ground facts e.g., TextRunner [Banko et. al., IJCAI’07] SNE Unsupervised, domain-independent Scales to Web Text ! simple semantic network Abundance of Web text ! KB

5. Motivation Background Semantic Network Extractor Experiments Future Work 5 Overview

6. Motivation Background Semantic Network Extractor Experiments Future Work 6 Overview

7. 7 Markov Logic A logical KB is a set of hard constraints on the set of possible worlds Let’s make them soft constraints: When a world violates a formula, it becomes less probable, not impossible Give each formula a weight (Higher weight  Stronger constraint)

8. 8 TextRunner [Banko et al. IJCAI’07] Extracts (object,relation,object) triples from webpages in a single pass Identify nouns with noun phrase chunker Heuristically identify string between two nouns as relation Classify each triple as true or false using Naïve Bayes classifier

9. 9 Motivation Background Semantic Network Extractor Experiments Future Work Overview

10. 10 Semantic Network Extractor Input: tuples r(x,y) Output: simple semantic network Clusters objects and relations simultaneously Number of clusters need not be specified in advance Cluster relations by objects they relate and vice versa

11. Cluster:,, Clustering:,, 11 Notation r2r2 r1r1 r3r3 r7r7 r6r6 r5r5 r4r4 x1x1 x2x2 x3x3 x4x4 x5x5 x6x6 y1y1 y2y2 y3y3 y4y4 y5y5

12. 12 Notation Atom:,,, Cluster combination: r2r2 r1r1 r3r3 r7r7 r6r6 r5r5 r4r4 x1x1 x2x2 x3x3 x4x4 x5x5 x6x6 y1y1 y2y2 y3y3 y4y4 y5y5

13. 13 SNE Model Four rules Each symbol belongs to exactly one cluster Exponential prior on #cluster combinations Most symbols tend to be in different clusters

14. 14 Atom prediction rule: Truth value of atom is determined by cluster combination it belongs to SNE Model Wt of rule is log-odds of atom in its cluster combination being true #true & #false atoms in cluster combination Smoothing parameters

15. 15 Learning SNE Model Learning consists of finding Weights of atom prediction rules Cluster assignment  (  r,  x,  y )  assignment of truth values to, and atoms vector of truth assignments to all observed ground atoms r(x,y) that maximize log-posterior probability first three rules atom prediction rule

16. 16 Log Posterior Set of cluster combinations #cluster combinations #pairs of symbols in different clusters constant prob. atom is trueprob. atom is false Intractable!

17. 17 Number of Cluster Combinations space_shuttle Columbia astronomers Earth planet Kennedy_Space_Center delivered_to orbits think_about

18. 18 Number of Cluster Combinations think_about astronomers Earth planet orbits space_shuttle Voyager Kennedy_Space_Center delivered_to space_shuttle Columbia astronomers earth planet Kennedy_Space_Center delivered_to orbits think_about

19. 19 Log-Posterior (Approximation) S i set of symbols of type i #false atoms in cluster comb. with only false atoms Pr(atom=false) Assume atoms in cluster combinations with only false atoms all belong to a single ‘default’ cluster combination #cluster comb. with ¸ 1 true r(x,y) atom Set of cluster comb. With ¸ 1 true r(x,y) atom

20. 20 Search Algorithm Approximation: Hard assignment of symbols to clusters Searches over cluster assignments, evaluate each by its log-posterior Agglomerative clustering Start with each r, x, y symbols in own cluster Merge pairs of clusters in bottom-up manner

21. orbits revolves_around orbits 21 Search Algorithm Apollo 10 Earth Odyssey space_shuttle Moon … … orbits revolves_around

22. Moon space_shuttle Apollo 10 space_shuttle Odyssey Apollo 10 Odyssey Apollo 10 Odyssey 22 Search Algorithm Earth … … orbits revolves_around orbits revolves_around Earth

23. orbits revolves_around 23 Search Algorithm Apollo 10 Earth Odyssey space_shuttle Moon … …

24. 24 Motivation Background Semantic Network Extractor Experiments Future Work Overview

25. 25 Dataset 2.1 million triples extracted in Web crawl by TextRunner [Banko et al, IJCAI 2007] e.g., named_after(Jupiter,Roman_god), upheld(Court,ruling), etc. 15,872 r symbols, 700,781 x symbols, 665,378 y symbols Only consider symbols appearing ¸ 25 times 10,214 r symbols, 8942 x symbols, 7995 y symbols 2,065,045 triples contain at least one such symbol

26. 26 Comparison Systems Multiple Relational Clustering (MRC) [Kok & Domingos, ICML’07] Similar to SNE Finds multiple clusterings Exponential prior on #clusters No symbols pairs tend to be in different clusters rule Information-Theoretic Co-clustering (ITC) [Dhillon et al., KDD’03] Clusters data in 2D matrix along both dimensions Maximize mutual info b/w row & column clusters Extended it to 3D Extended it to use BIC prior on #cluster combinations Infinite Relational Model (IRM) [Kemp et al., AAAI’06] Generative model: Beta ! p ! Bernoulli ! Atoms Changed it to use CRP prior #cluster combination Search algorithms changed to SNE’s agglomerative clustering

27. 27 Evaluation Pairwise precision, recall, & F1 against manually created gold standard 2688 r symbols, 2568 x symbols, 3058 y symbols assigned to non-unit clusters 874 r non-unit clusters, 511 x non-unit clusters, 700 y non-unit clusters Remaining symbols assigned to unit clusters Correct semantic statements Cluster combinations with ¸ 5 true ground r(x,y) atoms

28. 28 Parameter Settings Closed-world assumption triples not in DB are assumed false SNE parameters: =  = 100, p false = 0.9999  = 2.81 £ 10 -9,  = 10 - ,  /(  +  ) = fraction of true triples in dataset Tried various parameters values for MRC, ITC, and IRM, and chose the best ones

29. 29 SNE vs. MRC

30. 30 SNE vs. IRM vs. ITC

31. 31 SNE vs. ITC vs. IRM 0.778 0.874 0.835 >2x>3x

32. 32 SNE vs. ITC vs. IRM Hours

33. 33 SNE Full Joint Model vs. Separate Clustering

34. 34 SNE and WordNet Compare SNE’s object clusters with WordNet 5000 object symbols overlaps with WordNet Convert each node (synset) in WordNet taxonomy to contain children concepts too Match SNE cluster to WordNet cluster with best F1 score Lower the matched cluster in WordNet taxonomy, more precise the concept

35. 35 Levels of Matched WordNet Clusters Level 47 SNE Cluster Size 362419161211108765432

36. 36 Snippet of Extracted Semantic Network brother father mother parents couple family friends research studies study results America US Australia Austria Britain India Germany … Islam Judaism Catholicism Christianity Protestanism emigrated_to relocated_to escaped_to moved_back_to converted_to embraced to_embrace conducted_in carried_out_in

37. 37 Motivation Background Semantic Network Extractor Experiments Future Work Overview

38. 38 Future Work Integrate tuple extraction into SNE Learn richer semantic networks Learn logical theories Etc.

39. 39 Conclusion SNE: unsupervised, domain-independent approach, Text ! Simple semantic network Takes us a step closer to “grand agenda” of Text ! KB Based on Markov logic Techniques to scale SNE up to the Web Comparisons with other approaches show promise of SNE