1. Department of Computer Science The University of Texas at Austin USA Joint Entity and Relation Extraction using Card-Pyramid Parsing Rohit J. Kate Raymond J. Mooney

2. 2 Entity and Relation Extraction Information Extraction is the task of extracting structured information from text Entity Extraction Person, Location, Organization Relation Extraction Located_In(Location, Location) Work_For(Person, Organization) OrgBased_In(Organization, Location) Live_In(Person, Location) Kill(Person, Person)

3. 3 Entity and Relation Extraction Austin lives in Los Angeles, California and works there for an American company called ABC Inc. PersonLocation OtherOrganization Work_For OrgBased_In Live_In Located_In

4. 4 Entity and Relation Extraction Traditionally, entity and relation extraction is done in a pipeline First entities are extracted Then relations are extracted assuming that the extracted entities are correct

5. 5 Entity and Relation Extraction However, relations can influence entity extraction Austin lives in Los Angeles, California and works there for an American company called ABC Inc. Person? Location? Location Live_In Person

6. 6 Entity and Relation Extraction Relations can also influence extracting other relations Austin lives in Los Angeles, California and works there for an American company called ABC Inc. PersonLocation Organization Work_For OrgBased_In Live_In

7. 7 Joint Entity and Relation Extraction Both entity and relation extraction can benefit if done jointly –Correct errors of each other –Influence each other A brute force algorithm to find the most probable joint extraction is intractable –If there are n entities in a sentence then O(n 2 ) possible relations between them and for r relation labels O(r n^2 ) possibilities We present a new method for joint extraction

8. 8 Joint Entity and Relation Extraction Treat it analogous to parsing with the following productions: –Entity productions: Person  Candidate_entity Location  Candidate_entity Organization  Candidate_entity –Relation productions: Located_In  Location Location Work_For  Person Organization OrgBased_In  Organization Location Live_In  Person Location Kill  Person Person

9. 9 Joint Entity and Relation Extraction However, many entities are in multiple relations, with a lot of overlapping Context-free grammar (CFG) tree structure is not adequate We introduce a new structure we call card-pyramid Austin lives in Los Angeles, California and works there for an American company called ABC Inc. PersonLocation OtherOrganization Work_For OrgBased_In Live_In Located_In

10. 10 Joint Entity and Relation Extraction using Card-Pyramid Austin lives in Los Angeles, California and works there for an American company called ABC Inc. PersonLocation OtherOrganization Work_For OrgBased_In Live_In Located_In Person Location Other Organization (Austin)(Los Angeles)(California)(American)(ABC Inc) Work_For OrgBased_In Not_Related Candidate entities Live_In Located_In

11. 11 Joint Entity and Relation Extraction using Card-Pyramid Entities and their relations are compactly represented in a card-pyramid graph Joint entity and relation extraction reduces to finding the most probable joint labeling of its nodes We developed an efficient bottom-up card- pyramid parsing algorithm which uses dynamic programming and beam search, given entity and relation classifiers

12. 12 Distinction from CFG Tree No overlap Overlap CFG Tree Card-Pyramid

13. 13 Distinction from CFG Tree No overlap Overlap CFG Tree Card-Pyramid

14. 14 Card-Pyramid Parsing Assumes candidate entities are given –Can be obtained automatically [Punyakanok & Roth, 2001] –Use a simple heuristic, like all noun-phrase chunks –In the worst case include every substring, they will get label Other if they are none of the given types Austin lives in Los Angeles, California and works there for an American company called ABC Inc. (Austin)(Los Angeles)(California)(American)(ABC Inc)

15. 15 Card-Pyramid Parsing Austin lives in Los Angeles, California and works there for an American company called ABC Inc. (Austin)(Los Angeles)(California)(American)(ABC Inc) Apply entity classifiers at the leaf nodes SVM with standard features: words, POS tags, capitalization, gazetteer, suffixes etc. Loc0.9 Per0.8 Org0.1 Loc0.95 Org0.2 Oth0.01 Loc0.98 Per0.1 Oth0.02 Oth0.8 Per0.2 Org0.1 Org0.92 Loc0.21 Oth0.01 Beam Beam element

16. 16 Card-Pyramid Parsing Austin lives in Los Angeles, California and works there for an American company called ABC Inc. (Austin)(Los Angeles)(California)(American)(ABC Inc) Apply relation classifiers bottom-up at the internal nodes relating leftmost and rightmost leaves SVM with word subsequence kernel for before, between and after patterns of the two entities [Bunescu & Mooney, 2005] Loc0.9 Per0.8 Org0.1 Loc0.95 Org0.2 Oth0.01 Loc0.98 Per0.1 Oth0.02 Oth0.8 Per0.2 Org0.1 Org0.92 Loc0.21 Oth0.01

17. 17 Card-Pyramid Parsing Austin lives in Los Angeles, California and works there for an American company called ABC Inc. (Austin)(Los Angeles)(California)(American)(ABC Inc) Loc0.98 Per0.1 Oth0.02 Oth0.8 Per0.2 Org0.1 Org0.92 Loc0.21 Oth0.01 Loc0.9 Per0.8 Org0.1 Loc0.95 Org0.2 Oth0.01 Consider every combination of children’s beam elements

18. 18 Card-Pyramid Parsing Austin lives in Los Angeles, California and works there for an American company called ABC Inc. (Austin)(Los Angeles)(California)(American)(ABC Inc) Loc0.9 Per0.8 Org0.1 Loc0.95 Org0.2 Oth0.01 Loc0.98 Per0.1 Oth0.02 Oth0.8 Per0.2 Org0.1 Org0.92 Loc0.21 Oth0.01Work_For0.1 Work_For  Per Org

19. 19 Card-Pyramid Parsing Austin lives in Los Angeles, California and works there for an American company called ABC Inc. (Austin)(Los Angeles)(California)(American)(ABC Inc) Loc0.9 Per0.8 Org0.1 Loc0.95 Org0.2 Oth0.01 Loc0.98 Per0.1 Oth0.02 Oth0.8 Per0.2 Org0.1 Org0.92 Loc0.21 Oth0.01Work_For0.10.01 Work_For  Per Org

20. 20 Card-Pyramid Parsing Austin lives in Los Angeles, California and works there for an American company called ABC Inc. (Austin)(Los Angeles)(California)(American)(ABC Inc) Loc0.9 Per0.8 Org0.1 Loc0.95 Org0.2 Oth0.01 Loc0.98 Per0.1 Oth0.02 Oth0.8 Per0.2 Org0.1 Org0.92 Loc0.21 Oth0.01Work_For0.10.01

21. 21 Card-Pyramid Parsing Austin lives in Los Angeles, California and works there for an American company called ABC Inc. (Austin)(Los Angeles)(California)(American)(ABC Inc) Loc0.9 Per0.8 Org0.1 Loc0.95 Org0.2 Oth0.01 Loc0.98 Per0.1 Oth0.02 Oth0.8 Per0.2 Org0.1 Org0.92 Loc0.21 Oth0.01 Located_In0.10.08 Work_For0.10.01 Located_In  Loc Loc

22. 22 Card-Pyramid Parsing Austin lives in Los Angeles, California and works there for an American company called ABC Inc. (Austin)(Los Angeles)(California)(American)(ABC Inc) Loc0.9 Per0.8 Org0.1 Loc0.95 Org0.2 Oth0.01 Loc0.98 Per0.1 Oth0.02 Oth0.8 Per0.2 Org0.1 Org0.92 Loc0.21 Oth0.01 Located_In0.10.08 Work_For0.10.01

23. 23 Card-Pyramid Parsing Austin lives in Los Angeles, California and works there for an American company called ABC Inc. (Austin)(Los Angeles)(California)(American)(ABC Inc) Loc0.9 Per0.8 Org0.1 Loc0.95 Org0.2 Oth0.01 Loc0.98 Per0.1 Oth0.02 Oth0.8 Per0.2 Org0.1 Org0.92 Loc0.21 Oth0.01 Live_In0.90.68 Located_In0.10.08 Work_For0.10.01 Live_In  Per Loc

24. 24 Card-Pyramid Parsing Austin lives in Los Angeles, California and works there for an American company called ABC Inc. (Austin)(Los Angeles)(California)(American)(ABC Inc) Loc0.9 Per0.8 Org0.1 Loc0.95 Org0.2 Oth0.01 Loc0.98 Per0.1 Oth0.02 Oth0.8 Per0.2 Org0.1 Org0.92 Loc0.21 Oth0.01 Live_In0.90.68 Located_In0.10.08 Work_For0.10.01 Located_In0.90.80 OrgBased_In0.60.47 Not_Related0.50.00 …………..

25. 25 Card-Pyramid Parsing Austin lives in Los Angeles, California and works there for an American company called ABC Inc. (Austin)(Los Angeles)(California)(American)(ABC Inc) Loc0.9 Per0.8 Org0.1 Loc0.95 Org0.2 Oth0.01 Loc0.98 Per0.1 Oth0.02 Oth0.8 Per0.2 Org0.1 Org0.92 Loc0.21 Oth0.01 Live_In0.90.68 Located_In0.10.08 Work_For0.10.01 Located_In0.90.80 OrgBased_In0.60.47 Not_Related0.50.00 …………..

26. 26 Card-Pyramid Parsing Austin lives in Los Angeles, California and works there for an American company called ABC Inc. (Austin)(Los Angeles)(California)(American)(ABC Inc) Loc0.9 Per0.8 Org0.1 Loc0.95 Org0.2 Oth0.01 Loc0.98 Per0.1 Oth0.02 Oth0.8 Per0.2 Org0.1 Org0.92 Loc0.21 Oth0.01 Live_In0.90.68 Located_In0.10.08 Work_For0.10.01 Located_In0.90.80 OrgBased_In0.60.47 Not_Related0.50.00 …………..

27. 27 Card-Pyramid Parsing Austin lives in Los Angeles, California and works there for an American company called ABC Inc. (Austin)(Los Angeles)(California)(American)(ABC Inc) Loc0.9 Per0.8 Org0.1 Loc0.95 Org0.2 Oth0.01 Loc0.98 Per0.1 Oth0.02 Oth0.8 Per0.2 Org0.1 Org0.92 Loc0.21 Oth0.01 Live_In0.90.68 Located_In0.10.08 Work_For0.10.01 Located_In0.90.80 OrgBased_In0.60.47 Not_Related0.50.01 ………….. Live_In0.9 Live_In  Per Loc Relations with in-between entities are also used as features, for e.g. “Live_In -- Located_In” In general, any features can be used from the sub-card-pyramid underneath.

28. 28 Card-Pyramid Parsing Austin lives in Los Angeles, California and works there for an American company called ABC Inc. (Austin)(Los Angeles)(California)(American)(ABC Inc) Loc0.9 Per0.8 Org0.1 Loc0.95 Org0.2 Oth0.01 Loc0.98 Per0.1 Oth0.02 Oth0.8 Per0.2 Org0.1 Org0.92 Loc0.21 Oth0.01 Live_In0.90.68 Located_In0.10.08 Work_For0.10.01 Located_In0.900.80 OrgBased_In0.60.47 Not_Related0.50.00 ………….. Live_In0.90.48 Live_In  Per Loc Divide the probability which gets multiplied twice.

29. 29 Card-Pyramid Parsing Austin lives in Los Angeles, California and works there for an American company called ABC Inc. (Austin)(Los Angeles)(California)(American)(ABC Inc) Loc0.9 Per0.8 Org0.1 Loc0.95 Org0.2 Oth0.01 Loc0.98 Per0.1 Oth0.02 Oth0.8 Per0.2 Org0.1 Org0.92 Loc0.21 Oth0.01 Live_In0.90.68 Located_In0.10.02 Work_For0.10.01 Located_In0.900.80 OrgBased_In0.60.47 Not_Related0.50.01 ………….. Live_In0.90.48 A beam element represents a sub-card-pyramid.

30. 30 Card-Pyramid Parsing Austin lives in Los Angeles, California and works there for an American company called ABC Inc. (Austin)(Los Angeles)(California)(American)(ABC Inc) Loc0.9 Per0.8 Org0.1 Loc0.95 Org0.2 Oth0.01 Loc0.98 Per0.1 Oth0.02 Oth0.8 Per0.2 Org0.1 Org0.92 Loc0.21 Oth0.01 Live_In0.90.68 Located_In0.10.02 Work_For0.10.01 Located_In0.900.80 OrgBased_In0.60.47 Not_Related0.50.00 ………….. Live_In0.90.48

31. 31 Card-Pyramid Parsing Austin lives in Los Angeles, California and works there for an American company called ABC Inc. (Austin)(Los Angeles)(California)(American)(ABC Inc) Loc0.9 Per0.8 Org0.1 Loc0.95 Org0.2 Oth0.01 Loc0.98 Per0.1 Oth0.02 Oth0.8 Per0.2 Org0.1 Org0.92 Loc0.21 Oth0.01 Live_In0.90.68 Located_In0.10.08 Work_For0.10.01 Located_In0.900.80 OrgBased_In0.60.47 Not_Related0.50.00 ………….. Live_In0.90.48 X An O(1) check for consistency of overlap.

32. 32 Card-Pyramid Parsing Austin lives in Los Angeles, California and works there for an American company called ABC Inc. (Austin)(Los Angeles)(California)(American)(ABC Inc) Loc0.9 Per0.8 Org0.1 Loc0.95 Org0.2 Oth0.01 Loc0.98 Per0.1 Oth0.02 Oth0.8 Per0.2 Org0.1 Org0.92 Loc0.21 Oth0.01 Live_In0.90.68 Located_In0.10.08 Work_For0.10.01 Located_In0.900.80 OrgBased_In0.60.47 Not_Related0.50.00 ………….. Live_In0.90.48 X An O(1) check for consistency of overlap.

33. 33 Card-Pyramid Parsing Austin lives in Los Angeles, California and works there for an American company called ABC Inc. (Austin)(Los Angeles)(California)(American)(ABC Inc) Loc0.9 Per0.8 Org0.1 Loc0.95 Org0.2 Oth0.01 Loc0.98 Per0.1 Oth0.02 Oth0.8 Per0.2 Org0.1 Org0.92 Loc0.21 Oth0.01 Live_In0.90.68 Located_In0.10.08 Work_For0.10.01 Located_In0.900.80 OrgBased_In0.60.47 Not_Related0.50.00 ………….. Live_In0.90.48 Most probable card-pyramid is represented by the top beam element at the root. An approximation because of the finite beam size.

34. 34 Training Classifiers Obtain correctly labeled card-pyramids for the annotated sentences in the training data Collect the positive examples for all the classifiers from the labels of the card-pyramids Positive examples for an entity classifier become negative examples for all other entity classifiers Pairs of entities with correct entity types but not related by a relation become negative examples for that relation’s classifier

35. 35 Related Work Roth & Yih [2004, 2007] –Employs independent entity and relation classifiers –Uses linear programming to find a consistent global solution from the classifier outputs –Output of other classifiers can’t be used as features Riedel et al. [2009] –Solves a related problem of extracting bio-molecular events and their arguments using Markov Logic Network –Single joint probabilistic model –Restricts extractors’ learning algorithm to Markov Logic Network’s learning algorithm, for example, cannot use kernel- based SVM for relation extraction Kate & Mooney [2006] –Parse using a suite of classifiers to find the most probable semantic parse

36. 36 Experiments Dataset used by Roth & Yih [2004, 2007] Number of sentences: 1437 Entities: Person (1685), Location (1968), Organization (978), Other (705) Relation Extraction Located_In(Location, Location) (406) Work_For(Person, Organization) (394) OrgBased_In(Organization, Location) (451) Live_In(Person, Location) (521) Kill(Person, Person) (268) Not_Related (17007)

37. 37 Experiments Performed five-fold cross-validation Measured: –Precision (percentage of output labels correct) –Recall (percentage of gold-standard labels correctly identified) –F-measure (harmonic mean) Compared with: –A pipelined approach using our entity and relation classifiers –Best results of Roth & Yih [2007] on joint extraction

38. 38 Results: Entity Extraction F-measure

39. 39 Results: Relation Extraction F-measure An unusual sentence with 20 Locations separated by commas.

40. 40 A General Method to Extract Structured Information from a Sentence Encode what you want to extract and constraints between them in the productions Train a classifier for every production Apply the classifiers to find the most probable structure allowed by the productions to jointly find the structured information

41. 41 Future Work Extract higher order relations: relations between relations such as temporal or causal relations Jointly perform co-reference resolution with entity and relation extraction –Add a new production: Coref  Person Person Model the structure of card-pyramid using a probabilistic graphical model A kernel to compute similarity between two card- pyramids and use it for relation classifier

42. 42 Conclusions Introduced a card-pyramid structure for joint entity and relation extraction Compactly encode entities and relations in a sentence Joint extraction reduces to jointly labeling the nodes Presented an efficient parsing algorithm for joint labeling Experiments demonstrated benefits of the approach

43. 43 Thanks! Questions?