1. Information Extraction Junichi Tsujii Graduate School of Science University of Tokyo Japan Ronen Feldman Bar Ilan University Israel

2. Application Tasks of NLP (1)Information Retrieval/Detection (2)Passage Retrieval (3)Information Extraction (5)Text Understanding (4) Question/Answering Tasks To search and retrieve documents in response to queries for information To search and retrieve part of documents in response to queries for information To extract information that fits pre-defined database schemas or templates, specifying the output formats To answer general questions by using texts as knowledge base: Fact retrieval, combination of IR and IE To understand texts as people do: Artificial Intelligence

3. (1)Information Retrieval/Detection (2)Passage Retrieval (3)Information Extraction (5)Text Understanding (4) Question/Answering Tasks Ranges of Queries Pre-Defined: Fixed aspects of information carried in texts

4. IE definitions  Entity: an object of interest such as a person or organization  Attribute: A property of an entity such as name, alias, descriptor or type  Fact: A relationship held between two or more entities such as Position of Person in Company  Event: An activity involving several entities such as terrorist act, airline crash, product information

5. IE accuracy typical figures by information type  Entity: 90-98%  Attribute: 80%  Fact: 60-70%  Event:50-60%

6. MUC conferences  MUC 1 to MUC 7  1987 to 1997  Topics: Naval operations (2) Terrorist Activity (2) Joint venture and microelectronics Management changes Space Vehicles and Missile launches

7. The ACE Evaluation  The ACE program – challenge of extracting content from human language. Research effort directed to master first the extraction of “entities” Then the extraction of “relations” among these entities Finally the extraction of “events” that are causally related sets of relations  After two years, top systems successfully capture well over 50 % of the value at the entity level

8. Applications of IE  Routing of information  Infrastructure for IR and categorization (higher level features)  Event based summarization  Automatic creation of databases and knowledge bases

9. Where would IE be useful?  Semi-structured text  Generic documents like news articles  Most of the information in the doc is centred around a set of easily identifiable entities

10. Bridgestone Sports Co. said Friday it had set up a joint venture in Taiwan with a local concern and a Japanese trading house to produce golf clubs to be supplied to Japan. The joint venture, Bridgestone Sports Taiwan Co., capitalized at 20 million new Taiwan dollars, will start production in January 1990 with production of 20,000 iron and “metal wood” clubs a month. Example of IE: FASTUS(1993) TIE-UP-1 Relationship: TIE-UP Entities: “Bridgestone Sport Co.” “a local concern” “a Japanese trading house” Joint Venture Company: “Bridgestone Sports Taiwan Co.” Activity: ACTIVITY-1 Amount: NT$200000000 ACTIVITY-1 Activity: PRODUCTION Company: “Bridgestone Sports Taiwan Co.” Product: “iron and ‘metal wood’ clubs” Start Date: DURING: January 1990

11. Bridgestone Sports Co. said Friday it had set up a joint venture in Taiwan with a local concern and a Japanese trading house to produce golf clubs to be supplied to Japan. The joint venture, Bridgestone Sports Taiwan Co., capitalized at 20 million new Taiwan dollars, will start production in January 1990 with production of 20,000 iron and “metal wood” clubs a month. Example of IE: FASTUS(1993) TIE-UP-1 Relationship: TIE-UP Entities: “Bridgestone Sport Co.” “a local concern” “a Japanese trading house” Joint Venture Company: “Bridgestone Sports Taiwan Co.” Activity: ACTIVITY-1 Amount: NT$200000000 ACTIVITY-1 Activity: PRODUCTION Company: “Bridgestone Sports Taiwan Co.” Product: “iron and ‘metal wood’ clubs” Start Date: DURING: January 1990

12. Bridgestone Sports Co. said Friday it had set up a joint venture in Taiwan with a local concern and a Japanese trading house to produce golf clubs to be supplied to Japan. The joint venture, Bridgestone Sports Taiwan Co., capitalized at 20 million new Taiwan dollars, will start production in January 1990 with production of 20,000 iron and “metal wood” clubs a month. Example of IE: FASTUS(1993) TIE-UP-1 Relationship: TIE-UP Entities: “Bridgestone Sport Co.” “a local concern” “a Japanese trading house” Joint Venture Company: “Bridgestone Sports Taiwan Co.” Activity: ACTIVITY-1 Amount: NT$200000000 ACTIVITY-1 Activity: PRODUCTION Company: “Bridgestone Sports Taiwan Co.” Product: “iron and ‘metal wood’ clubs” Start Date: DURING: January 1990

13. Bridgestone Sports Co. said Friday it had set up a joint venture in Taiwan with a local concern and a Japanese trading house to produce golf clubs to be supplied to Japan. The joint venture, Bridgestone Sports Taiwan Co., capitalized at 20 million new Taiwan dollars, will start production in January 1990 with production of 20,000 iron and “metal wood” clubs a month. Example of IE: FASTUS(1993) TIE-UP-1 Relationship: TIE-UP Entities: “Bridgestone Sport Co.” “a local concern” “a Japanese trading house” Joint Venture Company: “Bridgestone Sports Taiwan Co.” Activity: ACTIVITY-1 Amount: NT$200000000 ACTIVITY-1 Activity: PRODUCTION Company: “Bridgestone Sports Taiwan Co.” Product: “iron and ‘metal wood’ clubs” Start Date: DURING: January 1990

14. Bridgestone Sports Co. said Friday it had set up a joint venture in Taiwan with a local concern and a Japanese trading house to produce golf clubs to be supplied to Japan. The joint venture, Bridgestone Sports Taiwan Co., capitalized at 20 million new Taiwan dollars, will start production in January 1990 with production of 20,000 iron and “metal wood” clubs a month. Example of IE: FASTUS(1993) TIE-UP-1 Relationship: TIE-UP Entities: “Bridgestone Sport Co.” “a local concern” “a Japanese trading house” Joint Venture Company: “Bridgestone Sports Taiwan Co.” Activity: ACTIVITY-1 Amount: NT$200000000 ACTIVITY-1 Activity: PRODUCTION Company: “Bridgestone Sports Taiwan Co.” Product: “iron and ‘metal wood’ clubs” Start Date: DURING: January 1990

15. FASTUS 1.Complex Words: Recognition of multi-words and proper names 2.Basic Phrases: Simple noun groups, verb groups and particles 3.Complex phrases: Complex noun groups and verb groups 4.Domain Events: Patterns for events of interest to the application Basic templates are to be built. 5. Merging Structures: Templates from different parts of the texts are merged if they provide information about the same entity or event. Based on finite states automata (FSA) set up new Twaiwan dallors a Japanese trading house had set up production of 20, 000 iron and metal wood clubs [company] [set up] [Joint-Venture] with [company]

16. Bridgestone Sports Co. said Friday it had set up a joint venture in Taiwan with a local concern and a Japanese trading house to produce golf clubs to be supplied to Japan. The joint venture, Bridgestone Sports Taiwan Co., capitalized at 20 million new Taiwan dollars, will start production in January 1990 with production of 20,000 iron and “metal wood” clubs a month. Example of IE: FASTUS(1993) TIE-UP-1 Relationship: TIE-UP Entities: “Bridgestone Sport Co.” “a local concern” “a Japanese trading house” Joint Venture Company: “Bridgestone Sports Taiwan Co.” Activity: ACTIVITY-1 Amount: NT$200000000 ACTIVITY-1 Activity: PRODUCTION Company: “Bridgestone Sports Taiwan Co.” Product: “iron and ‘metal wood’ clubs” Start Date: DURING: January 1990

17. Information Extraction ………. Jurgen Pfrang, 51, reportedly stumbled upon the robbers on the second floor of his Nanjing home early on Sunday. The deputy general manager of Yaxing Benz, a Sino-German joint venture that makes buses and bus chassis in nearby Yangzhou, was hacked to death with 45 cm watermelon knives. ………. Name of the Venture: Yaxing Benz Products: buses and bus chassis Location: Yangzhou,China Companies involved: (1)Name: X? Country: German (2)Name: Y? Country: China

18. Information Extraction A German vehicle-firm executive was stabbed to death …. ………. Jurgen Pfrang, 51, reportedly stumbled upon the robbers on the second floor of his Nanjing home early on Sunday. The deputy general manager of Yaxing Benz, a Sino-German joint venture that makes buses and bus chassis in nearby Yangzhou, was hacked to death with 45 cm watermelon knives. ………. Crime-Type: Murder Type: Stabbing The killed: Name: Jurgen Pfrang Age: 51 Profession: Deputy general manager Location: Nanjing, China Different template for crimes

19. (1)Information Retrieval/Detection (2)Passage Retrieval (3)Information Extraction (5)Text Understanding (4) Question/Answering Tasks Interpretation of Texts User System

20. Collection of Texts IR System Characterization of Texts Queries

21. Collection of Texts IR System Characterization of Texts Queries Interpretation Knowledge

22. Collection of Texts Passage IR System Characterization of Texts Queries Interpretation Knowledge

23. Collection of Texts Passage IR System Characterization of Texts Queries Interpretation Knowledge IE System Texts Templates Structures of Sentences NLP

24. Interpretation Knowledge IE System Texts Templates

25. Interpretation Knowledge IE System Texts Templates Predefined General Framework of NLP/NLU IE as compromise NLP

26. (1)Information Retrieval/Detection (2)Passage Retrieval (3)Information Extraction (5)Text Understanding (4) Question/Answering Tasks Performance Evaluation Rather clear A bit vague Rather clear A bit vague Very vague

27. N N: Correct Documents M:Retrieved Documents C: Correct Documents that are actually retrieved M C Query Collection of Documents Precision: Recall: C M C N F-Value: P R P+R 2P ・ R

28. N N: Correct Templates M:Retrieved Templates C: Correct Templates that are actually retrieved M C Query Collection of Documents Precision: Recall: C M C N F-Value: P R P+R 2P ・ R More complicated due to partially filled templates

29. Framework of IE IE as compromise NLP

30. Syntactic Analysis General Framework of NLP Morphological and Lexical Processing Semantic Analysis Context processing Interpretation Difficulties of NLP (1) Robustness: Incomplete Knowledge Incomplete Domain Knowledge Interpretation Rules Predefined Aspects of Information

31. Syntactic Analysis General Framework of NLP Morphological and Lexical Processing Semantic Analysis Context processing Interpretation Difficulties of NLP (1) Robustness: Incomplete Knowledge Incomplete Domain Knowledge Interpretation Rules Predefined Aspects of Information

32. Approaches for building IE systems  Knowledge Engineering Approach Rules crafted by linguists in cooperation with domain experts Most of the work done by insoecting a set of relevant documents

33. Approaches for building IE systems  Automatically trainable systems Techniques based on statistics and almost no linguistic knowledge Language independent Main input – annotated corpus Small effort for creating rules, but crating annotated corpus laborious

34. Techniques in IE (1) Domain Specific Partial Knowledge: Knowledge relevant to information to be extracted (2) Ambiguities: Ignoring irrelevant ambiguities Simpler NLP techniques (4) Adaptation Techniques: Machine Learning, Trainable systems (3) Robustness: Coping with Incomplete dictionaries (open class words) Ignoring irrelevant parts of sentences

35. General Framework of NLP Morphological and Lexical Processing Syntactic Analysis Semantic Anaysis Context processing Interpretation Open class words: Named entity recognition (ex) Locations Persons Companies Organizations Position names Domain specific rules:, Inc. Mr.. Machine Learning: HMM, Decision Trees Rules + Machine Learning Part of Speech Tagger FSA rules Statistic taggers 95 % F-Value 90 Domain Dependent Local Context Statistical Bias

36. General Framework of NLP Morphological and Lexical Processing Syntactic Analysis Semantic Anaysis Context processing Interpretation FASTUS 1.Complex Words: Recognition of multi-words and proper names 2.Basic Phrases: Simple noun groups, verb groups and particles 3.Complex phrases: Complex noun groups and verb groups 4.Domain Events: Patterns for events of interest to the application Basic templates are to be built. 5. Merging Structures: Templates from different parts of the texts are merged if they provide information about the same entity or event. Based on finite states automata (FSA)

37. General Framework of NLP Morphological and Lexical Processing Syntactic Analysis Semantic Anaysis Context processing Interpretation FASTUS 1.Complex Words: Recognition of multi-words and proper names 2.Basic Phrases: Simple noun groups, verb groups and particles 3.Complex phrases: Complex noun groups and verb groups 4.Domain Events: Patterns for events of interest to the application Basic templates are to be built. 5. Merging Structures: Templates from different parts of the texts are merged if they provide information about the same entity or event. Based on finite states automata (FSA)

38. General Framework of NLP Morphological and Lexical Processing Syntactic Analysis Semantic Analysis Context processing Interpretation FASTUS 1.Complex Words: Recognition of multi-words and proper names 2.Basic Phrases: Simple noun groups, verb groups and particles 3.Complex phrases: Complex noun groups and verb groups 4.Domain Events: Patterns for events of interest to the application Basic templates are to be built. 5. Merging Structures: Templates from different parts of the texts are merged if they provide information about the same entity or event. Based on finite states automata (FSA)

39. Chomsky Hierarchy Hierarchy of Grammar of Automata Regular Grammar Finite State Automata Context Free Grammar Push Down Automata Context Sensitive Grammar Linear Bounded Automata Type 0 Grammar Turing Machine Computationally more complex, Less Efficiency

40. Chomsky Hierarchy Hierarchy of Grammar of Automata Regular Grammar Finite State Automata Context Free Grammar Push Down Automata Context Sensitive Grammar Linear Bounded Automata Type 0 Grammar Turing Machine Computationally more complex, Less Efficiency A B nn

41. 0 1 2 3 4 PN ’s ADJ Art N PN P ’s Art John’s interesting book with a nice cover

42. 0 1 2 3 4 PN ’s ADJ Art N PN P ’s Art John’s interesting book with a nice cover

43. 0 1 2 3 4 PN ’s ADJ Art N PN P ’s Art John’s interesting book with a nice cover

44. 0 1 2 3 4 PN ’s ADJ Art N PN P ’s Art John’s interesting book with a nice cover

45. 0 1 2 3 4 PN ’s ADJ Art N PN P ’s Art John’s interesting book with a nice cover

46. 0 1 2 3 4 PN ’s ADJ Art N PN P ’s Art John’s interesting book with a nice cover

47. 0 1 2 3 4 PN ’s ADJ Art N PN P ’s Art John’s interesting book with a nice cover

48. 0 1 2 3 4 PN ’s ADJ Art N PN P ’s Art John’s interesting book with a nice cover

49. 0 1 2 3 4 PN ’s ADJ Art N PN P ’s Art John’s interesting book with a nice cover

50. 0 1 2 3 4 PN ’s ADJ Art N PN P ’s Art John’s interesting book with a nice cover

51. 0 1 2 3 4 PN ’s ADJ Art N PN P ’s Art John’s interesting book with a nice cover Pattern-maching PN ’s (ADJ)* N P Art (ADJ)* N {PN ’s/ Art}(ADJ)* N(P Art (ADJ)* N)*

52. General Framework of NLP Morphological and Lexical Processing Syntactic Analysis Semantic Analysis Context processing Interpretation FASTUS 1.Complex Words: Recognition of multi-words and proper names 2.Basic Phrases: Simple noun groups, verb groups and particles 3.Complex phrases: Complex noun groups and verb groups 4.Domain Events: Patterns for events of interest to the application Basic templates are to be built. 5. Merging Structures: Templates from different parts of the texts are merged if they provide information about the same entity or event. Based on finite states automata (FSA)

53. Bridgestone Sports Co. said Friday it had set up a joint venture in Taiwan with a local concern and a Japanese trading house to produce golf clubs to be supplied to Japan. The joint venture, Bridgestone Sports Taiwan Co., capitalized at 20 million new Taiwan dollars, will start production in January 1990 with production of 20,000 “metal wood” clubs a month. Example of IE: FASTUS(1993) 1.Complex words 2.Basic Phrases: Bridgestone Sports Co.: Company name said : Verb Group Friday : Noun Group it : Noun Group had set up : Verb Group a joint venture : Noun Group in : Preposition Taiwan : Location Attachment Ambiguities are not made explicit

54. Bridgestone Sports Co. said Friday it had set up a joint venture in Taiwan with a local concern and a Japanese trading house to produce golf clubs to be supplied to Japan. The joint venture, Bridgestone Sports Taiwan Co., capitalized at 20 million new Taiwan dollars, will start production in January 1990 with production of 20,000 “metal wood” clubs a month. Example of IE: FASTUS(1993) 1.Complex words 2.Basic Phrases: Bridgestone Sports Co.: Company name said : Verb Group Friday : Noun Group it : Noun Group had set up : Verb Group a joint venture : Noun Group in : Preposition Taiwan : Location {{}} a Japanese tea house a [Japanese tea] house a Japanese [tea house]

55. Bridgestone Sports Co. said Friday it had set up a joint venture in Taiwan with a local concern and a Japanese trading house to produce golf clubs to be supplied to Japan. The joint venture, Bridgestone Sports Taiwan Co., capitalized at 20 million new Taiwan dollars, will start production in January 1990 with production of 20,000 “metal wood” clubs a month. Example of IE: FASTUS(1993) 1.Complex words 2.Basic Phrases: Bridgestone Sports Co.: Company name said : Verb Group Friday : Noun Group it : Noun Group had set up : Verb Group a joint venture : Noun Group in : Preposition Taiwan : Location Structural Ambiguities of NP are ignored

56. Bridgestone Sports Co. said Friday it had set up a joint venture in Taiwan with a local concern and a Japanese trading house to produce golf clubs to be supplied to Japan. The joint venture, Bridgestone Sports Taiwan Co., capitalized at 20 million new Taiwan dollars, will start production in January 1990 with production of 20,000 “metal wood” clubs a month. Example of IE: FASTUS(1993) 2.Basic Phrases: Bridgestone Sports Co.: Company name said : Verb Group Friday : Noun Group it : Noun Group had set up : Verb Group a joint venture : Noun Group in : Preposition Taiwan : Location 3.Complex Phrases

57. [COMPNY] said Friday it [SET-UP] [JOINT-VENTURE] in [LOCATION] with [COMPANY] and [COMPNY] to produce [PRODUCT] to be supplied to [LOCATION]. [JOINT-VENTURE], [COMPNY], capitalized at 20 million [CURRENCY-UNIT] [START] production in [TIME] with production of 20,000 [PRODUCT] a month. Example of IE: FASTUS(1993) 2.Basic Phrases: Bridgestone Sports Co.: Company name said : Verb Group Friday : Noun Group it : Noun Group had set up : Verb Group a joint venture : Noun Group in : Preposition Taiwan : Location 3.Complex Phrases Some syntactic structures like …

58. [COMPNY] said Friday it [SET-UP] [JOINT-VENTURE] in [LOCATION] with [COMPANY] to produce [PRODUCT] to be supplied to [LOCATION]. [JOINT-VENTURE] capitalized at [CURRENCY] [START] production in [TIME] with production of [PRODUCT] a month. Example of IE: FASTUS(1993) 2.Basic Phrases: Bridgestone Sports Co.: Company name said : Verb Group Friday : Noun Group it : Noun Group had set up : Verb Group a joint venture : Noun Group in : Preposition Taiwan : Location 3.Complex Phrases Syntactic structures relevant to information to be extracted are dealt with.

59. Syntactic variations GM set up a joint venture with Toyota. GM announced it was setting up a joint venture with Toyota. GM signed an agreement setting up a joint venture with Toyota. GM announced it was signing an agreement to set up a joint venture with Toyota.

60. Syntactic variations GM set up a joint venture with Toyota. GM announced it was setting up a joint venture with Toyota. GM signed an agreement setting up a joint venture with Toyota. GM announced it was signing an agreement to set up a joint venture with Toyota. [SET-UP] GM plans to set up a joint venture with Toyota. GM expects to set up a joint venture with Toyota. S NPVP VNP N VP V GM signed agreement setting up

61. Syntactic variations GM set up a joint venture with Toyota. GM announced it was setting up a joint venture with Toyota. GM signed an agreement setting up a joint venture with Toyota. GM announced it was signing an agreement to set up a joint venture with Toyota. [SET-UP] GM plans to set up a joint venture with Toyota. GM expects to set up a joint venture with Toyota. S NPVP V GM set up

62. [COMPNY] [SET-UP] [JOINT-VENTURE] in [LOCATION] with [COMPANY] to produce [PRODUCT] to be supplied to [LOCATION]. [JOINT-VENTURE] capitalized at [CURRENCY] [START] production in [TIME] with production of [PRODUCT] a month. Example of IE: FASTUS(1993) 3.Complex Phrases 4.Domain Events [COMPANY][SET-UP][JOINT-VENTURE]with[COMPNY] [COMPANY][SET-UP][JOINT-VENTURE] (others)* with[COMPNY] The attachment positions of PP are determined at this stage. Irrelevant parts of sentences are ignored.

63. Complications caused by syntactic variations Relative clause The mayor, who was kidnapped yesterday, was found dead today. [NG] Relpro {NG/others}* [VG] {NG/others}*[VG] [NG] Relpro {NG/others}* [VG]

64. Complications caused by syntactic variations Relative clause The mayor, who was kidnapped yesterday, was found dead today. [NG] Relpro {NG/others}* [VG] {NG/others}*[VG] [NG] Relpro {NG/others}* [VG]

65. Complications caused by syntactic variations Relative clause The mayor, who was kidnapped yesterday, was found dead today. [NG] Relpro {NG/others}* [VG] {NG/others}*[VG] [NG] Relpro {NG/others}* [VG] Basic patterns Surface Pattern Generator Patterns used by Domain Event Relative clause construction Passivization, etc.

66. FASTUS 1.Complex Words: 2.Basic Phrases: 3.Complex phrases: 4.Domain Events: Patterns for events of interest to the application Basic templates are to be built. 5. Merging Structures: Templates from different parts of the texts are merged if they provide information about the same entity or event. Based on finite states automata (FSA) Piece-wise recognition of basic templates Reconstructing information carried via syntactic structures by merging basic templates NP, who was kidnapped, was found.

67. FASTUS 1.Complex Words: 2.Basic Phrases: 3.Complex phrases: 4.Domain Events: Patterns for events of interest to the application Basic templates are to be built. 5. Merging Structures: Templates from different parts of the texts are merged if they provide information about the same entity or event. Based on finite states automata (FSA) Piece-wise recognition of basic templates Reconstructing information carried via syntactic structures by merging basic templates NP, who was kidnapped, was found.

68. FASTUS 1.Complex Words: 2.Basic Phrases: 3.Complex phrases: 4.Domain Events: Patterns for events of interest to the application Basic templates are to be built. 5. Merging Structures: Templates from different parts of the texts are merged if they provide information about the same entity or event. Based on finite states automata (FSA) Piece-wise recognition of basic templates Reconstructing information carried via syntactic structures by merging basic templates NP, who was kidnapped, was found.

69. FASTUS 1.Complex Words: 2.Basic Phrases: 3.Complex phrases: 4.Domain Events: Patterns for events of interest to the application Basic templates are to be built. 5. Merging Structures: Templates from different parts of the texts are merged if they provide information about the same entity or event. Based on finite states automata (FSA) Piece-wise recognition of basic templates Reconstructing information carried via syntactic structures by merging basic templates NP, who was kidnapped, was found.

70. Current state of the arts of IE 1.Carefully constructed IE systems F-60 level (interannotater agreement: 60-80%) Domain: telegraphic messages about naval operation (MUC-1:87, MUC-2:89) news articles and transcriptions of radio broadcasts Latin American terrorism (MUC-3:91, MUC-4:1992) News articles about joint ventures (MUC-5, 93) News articles about management changes (MUC-6, 95) News articles about space vehicle (MUC-7, 97) 2.Handcrafted rules (named entity recognition, domain events, etc) Automatic learning from texts: Supervised learning : corpus preparation Non-supervised, or controlled learning