Presentation is loading. Please wait.

Presentation is loading. Please wait.

Snowball : Extracting Relations from Large Plain-Text Collections

Published byLester Byrd Modified over 9 years ago

Similar presentations

Presentation on theme: "Snowball : Extracting Relations from Large Plain-Text Collections"— Presentation transcript:

1 Snowball : Extracting Relations from Large Plain-Text Collections
Eugene Agichtein Luis Gravano Department of Computer Science Columbia University

2 Extracting Relations from Documents
Text documents hide valuable structured information. If we manage to extract this information: We can answer user queries more accurately We can run data mining tasks (e.g., finding trends) Eugene Agichtein Columbia University

3 GOAL: Extract All Tuples “Hidden” in the Document Collection
System must: Require minimal training for each new task Recover from noise Exploit redundancy of information in documents Eugene Agichtein Columbia University

4 Example Task: Organization/Location
Redundancy Organization Location Microsoft's central headquarters in Redmond is home to almost every product group and division. Microsoft Apple Computer Nike Redmond Cupertino Portland Brent Barlow, 27, a software analyst and beta-tester at Apple Computer headquarters in Cupertino, was fired Monday for "thinking a little too different." Apple's programmers "think different" on a "campus" in Cupertino, Cal. Nike employees "just do it" at what the company refers to as its "World Campus," near Portland, Ore. Eugene Agichtein Columbia University

5 Extracting Relations from Text Collections
Related Work The Snowball System Evaluation Metrics Experimental Results Eugene Agichtein Columbia University

6 Eugene Agichtein Columbia University
Related Work Traditional Information Extraction MUCs (Message Understanding Conferences) Significant (manual) training for each new task Bootstrapping Riloff et al. (‘99), Collins & Singer (‘99) (Named-entity recognition) Brin (DIPRE) (‘98) Eugene Agichtein Columbia University

7 Extracting Relations from Text: DIPRE
Initial Seed Tuples: Initial Seed Tuples Occurrences of Seed Tuples Generate New Seed Tuples Augment Table Generate Extraction Patterns Eugene Agichtein Columbia University

8 Extracting Relations from Text: DIPRE
Computer servers at Microsoft’s headquarters in Redmond… In mid-afternoon trading, share of Redmond-based Microsoft fell… The Armonk-based IBM introduced a new line… The combined company will operate from Boeing’s headquarters in Seattle. Intel, Santa Clara, cut prices of its Pentium processor. Occurrences of seed tuples: Initial Seed Tuples Occurrences of Seed Tuples Generate New Seed Tuples Augment Table Generate Extraction Patterns Eugene Agichtein Columbia University

9 Extracting Relations from Text: DIPRE
<STRING1>’s headquarters in <STRING2> <STRING2> -based <STRING1> <STRING1> , <STRING2> DIPRE Patterns: Initial Seed Tuples Occurrences of Seed Tuples Generate New Seed Tuples Augment Table Generate Extraction Patterns Eugene Agichtein Columbia University

10 Extracting Relations from Text: DIPRE
Generate new seed tuples; start new iteration Initial Seed Tuples Occurrences of Seed Tuples Generate New Seed Tuples Augment Table Generate Extraction Patterns Eugene Agichtein Columbia University

11 Extracting Relations from Text: Potential Pitfalls
Invalid tuples generated Degrade quality of tuples on subsequent iterations Must have automatic way to select high quality tuples to use as new seed Pattern representation Patterns must generalize Eugene Agichtein Columbia University

12 Extracting Relations from Text Collections
Related Work DIPRE The Snowball System: Pattern representation and generation Tuple generation Automatic pattern and tuple evaluation Evaluation Metrics Experimental Results Eugene Agichtein Columbia University

13 Extracting Relations from Text: Snowball
Initial Seed Tuples: Initial Seed Tuples Occurrences of Seed Tuples Generate New Seed Tuples Tag Entities Augment Table Generate Extraction Patterns Eugene Agichtein Columbia University

14 Extracting Relations from Text: Snowball
Computer servers at Microsoft’s headquarters in Redmond… In mid-afternoon trading, share of Redmond-based Microsoft fell… The Armonk-based IBM introduced a new line… The combined company will operate from Boeing’s headquarters in Seattle. Intel, Santa Clara, cut prices of its Pentium processor. Occurrences of seed tuples: Initial Seed Tuples Occurrences of Seed Tuples Generate New Seed Tuples Tag Entities Augment Table Generate Extraction Patterns Eugene Agichtein Columbia University

15 Eugene Agichtein Columbia University
Problem: Patterns Excessively General Pattern: <STRING2>-based <STRING1> Today's merger with McDonnell Douglas positions Seattle -based Boeing to make major money in space. …, a producer of apple-based jelly, ... Incorrect! <jelly, apple> Eugene Agichtein Columbia University

16 Extracting Relations from Text: Snowball
Computer servers at Microsoft’s headquarters in Redmond… In mid-afternoon trading, share of Redmond-based Microsoft fell… The Armonk-based IBM introduced a new line… The combined company will operate from Boeing’s headquarters in Seattle. Intel, Santa Clara, cut prices of its Pentium processor. Tag Entities Use MITRE’s Alembic Named Entity tagger Initial Seed Tuples Occurrences of Seed Tuples Generate New Seed Tuples Tag Entities Augment Table Generate Extraction Patterns Eugene Agichtein Columbia University

17 Extracting Relations from Text
<ORGANIZATION>’s headquarters in <LOCATION> <LOCATION> -based <ORGANIZATION> <ORGANIZATION> , <LOCATION> PROBLEM: Patterns too specific: have to match text exactly. Initial Seed Tuples Occurrences of Seed Tuples Generate New Seed Tuples Tag Entities Augment Table Generate Extraction Patterns Eugene Agichtein Columbia University

18 Snowball: Pattern Representation
A Snowball pattern vector is a 5-tuple <left, tag1, middle, tag2, right>, tag1, tag2 are named-entity tags left, middle, and right are vectors of weighed terms. ORGANIZATION 's central headquarters in LOCATION is home to... {<'s 0.5>, <central 0.5> <headquarters 0.5>, < in 0.5>} {<is 0.75>, <home 0.75> } LOCATION ORGANIZATION < left , tag1 , middle , tag2 , right > Eugene Agichtein Columbia University

19 Snowball: Pattern Generation
Tagged Occurrences of seed tuples: Computer servers at Microsoft’s central headquarters in Redmond… In mid-afternoon trading, share of Redmond-based Microsoft fell… The Armonk -based IBM introduced a new line… The combined company will operate from Boeing’s headquarters in Seattle. Eugene Agichtein Columbia University

20 Snowball Pattern Generation: Cluster Similar Occurrences
Occurrences of seed tuples converted to Snowball representation: {<servers 0.75> <at 0.75>} {<’s 0.5> <central 0.5> <headquarters 0.5> <in 0.5>} LOCATION ORGANIZATION {<shares 0.75> <of 0.75>} {<- 0.75> <based 0.75> } {<fell 1>} LOCATION ORGANIZATION {<the 1>} {<- 0.75> <based 0.75> } {<introduced 0.75> <a 0.75>} LOCATION ORGANIZATION {<operate 0.75> <from 0.75>} {<’s 0.7> <headquarters 0.7> <in 0.7>} LOCATION ORGANIZATION Eugene Agichtein Columbia University

21 Eugene Agichtein Columbia University
Similarity Metric P = < Lp , tag1 , Mp , tag2 , Rp > S = < Ls , tag1 , Ms , tag2 , Rs > Match(P, S) = { Lp . Ls + Mp . Ms + Rp . Rs if the tags match otherwise Eugene Agichtein Columbia University

22 Snowball Pattern Generation: Clustering
{<servers 0.75> <at 0.75>} {<’s 0.5> <central 0.5> <headquarters 0.5> <in 0.5>} LOCATION ORGANIZATION {<operate 0.75> <from 0.75>} {<’s 0.7> <headquarters 0.7> <in 0.7>} LOCATION ORGANIZATION Cluster 2 {<shares 0.75> <of 0.75>} {<- 0.75> <based 0.75> } {<fell 1>} LOCATION ORGANIZATION {<the 1>} {<- 0.75> <based 0.75> } {<introduced 0.75> <a 0.75>} LOCATION ORGANIZATION Eugene Agichtein Columbia University

23 Snowball: Pattern Generation
Patterns are formed as centroids of the clusters. Filtered by minimum number of supporting tuples. {<’s 0.7> <in 0.7> <headquarters 0.7>} LOCATION Pattern1 ORGANIZATION {<- 0.75> <based 0.75>} LOCATION ORGANIZATION Pattern2 Eugene Agichtein Columbia University

24 Snowball: Tuple Extraction
Using the patterns, scan the collection to generate new seed tuples: Initial Seed Tuples Occurrences of Seed Tuples Generate New Seed Tuples Tag Entities Augment Table Generate Extraction Patterns Eugene Agichtein Columbia University

25 Snowball: Tuple Extraction
Represent each new text segment in the collection as the context 5-tuple: Find most similar pattern (if any) Netscape 's flashy headquarters in Mountain View is near {<'s 0.5>, <flashy 0.5>, <headquarters 0.5>, < in 0.5>} LOCATION {<is 0.75>, <near 0.75> } ORGANIZATION {<'s 0.7>, <headquarters 0.7>, < in 0.7>} LOCATION ORGANIZATION Eugene Agichtein Columbia University

26 Snowball: Automatic Pattern Evaluation
Seed tuples Pattern “ORGANIZATION, LOCATION” in action: Boeing, Seattle, said… Positive Intel, Santa Clara, cut prices… Positive invest in Microsoft, New York-based Negative analyst Jane Smith said Automatically estimate probability of a pattern generating valid tuples: Conf(Pattern) = _____Positive____ Positive + Negative e.g., Conf(Pattern) = 2/3 = 66% Pattern Confidence: Eugene Agichtein Columbia University

27 Snowball: Automatic Tuple Evaluation
Brent Barlow, 27, a software analyst and beta-tester at Apple Computer headquarters in Cupertino, was fired Monday for "thinking a little too different." <Apple Computer, Cupertino> Apple's programmers "think different" on a "campus" in Cupertino, Cal. Conf(Tuple) = 1 - (1 -Conf(Pi)) Estimation of Probability (Correct (Tuple) ) A tuple will have high confidence if generated by multiple high-confidence patterns (Pi). Eugene Agichtein Columbia University

28 Snowball: Filtering Seed Tuples
Generate new seed tuples: Initial Seed Tuples Occurrences of Seed Tuples Generate New Seed Tuples Tag Entities Augment Table Generate Extraction Patterns Eugene Agichtein Columbia University

29 Extracting Relations from Text Collections
Related Work The Snowball System: Pattern representation and generation Tuple generation Automatic pattern and tuple evaluation Evaluation Metrics Experimental Results Eugene Agichtein Columbia University

30 Task Evaluation Methodology
Data: Large collection, extracted tables contain many tuples (> 80,000) Need scalable methodology: Ideal set of tuples Automatic recall/precision estimation Estimated precision using sampling Eugene Agichtein Columbia University

31 Collections used in Experiments
More than 300,000 real newspaper articles Eugene Agichtein Columbia University

32 Eugene Agichtein Columbia University
The Ideal Metric (1) Creating the Ideal set of tuples All tuples mentioned in the collection Hoover’s directory (13K+ organizations) *Ideal * A perfect, (ideal) system would be able to extract all these tuples Eugene Agichtein Columbia University

33 Eugene Agichtein Columbia University
The Ideal Metric (2) Correct location found Extracted Ideal Precision: | Correct (Extracted  Ideal) | | Extracted  Ideal | Recall: | Correct (Extracted  Ideal) | | Ideal | Eugene Agichtein Columbia University

34 Estimate Precision by Sampling
Sample extracted table Random samples, each 100 tuples Manually check validity of tuples in each sample Eugene Agichtein Columbia University

35 Extracting Relations from Text Collections
Related Work The Snowball System: Pattern representation and generation Tuple generation Automatic pattern and tuple validation Evaluation Metrics Experimental Results Eugene Agichtein Columbia University

36 Experimental results: Test Collection
(b) (a) Recall (a) and precision (a) using the Ideal metric, plotted against the minimal number of occurrences of test tuples in the collection Eugene Agichtein Columbia University

37 Experimental results: Sample and Check
(b) Recall (a) and precision (b) for varying minimum confidence threshold Tt. NOTE: Recall is estimated using the Ideal metric, precision is estimated by manually checking random samples of result table. Eugene Agichtein Columbia University

38 Eugene Agichtein Columbia University
Conclusions We presented Our Snowball system: Requires minimal training (handful of seed tuples) Uses a flexible pattern representation Achieves high recall/precision > 80% of test tuples extracted Scalable evaluation methodology Eugene Agichtein Columbia University

39 Eugene Agichtein Columbia University
Recent and Future Work Recent (presented in DMKD’00 workshop) Alternative pattern representation Combining representations Future Work Evaluation on other extraction tasks Extensions: Non-binary relations Relations with no key  HTML documents Eugene Agichtein Columbia University

40 Snowball: Extracting Relations from Large Plain-Text Collections
Eugene Agichtein Luis Gravano Department of Computer Science Columbia University

41 Backup Slides

42 Eugene Agichtein Columbia University
Snowball Solutions Flexible pattern representation Pattern generation Automatic pattern and tuple evaluation Able to recover from noise Keeps only high quality tuples as new seed Eugene Agichtein Columbia University

43 Experimental Results: Training
(b) Recall (a) and precision (b) using the Ideal metric (training collection) Eugene Agichtein Columbia University

44 Sampling Results: Error Analysis
The tuples in the random samples were checked by hand to pinpoint the “culprits” responsible for incorrect tuples. Sample size is 100. Eugene Agichtein Columbia University

45 Sample Discovered Patterns
Eugene Agichtein Columbia University

46 Convergence of Snowball and DIPRE
Precision (a) and Recall (b) of the DIPRE and Snowball with increased iterations Eugene Agichtein Columbia University

47 Approximate Matching of Organizations
Use Whirl (W. AT&T) to match similar organization names Self-join the Extracted table on the Organization attribute Join resulting table with the Test table, and compare values of Location attributes Extracted Extracted ‘ Eugene Agichtein Columbia University

48 Eugene Agichtein Columbia University
References Blum & Mitchell. Combining labeled and unlabeled data with co-training. Proceedings of 1998 Conference on Computational Learning Theory. Brin. Extracting patterns and relations from the World-Wide Web. Proceedings on the 1998 International Workshop on Web and Databases (WebDB’98). Collins & Singer. Unsupervised models for named entity classification. EMNLP 1999. Riloff & Jones. Learning dictionaries for information extraction by multi-level bootstrapping. AAAI’99. Yarowsky. Unsupervised word sense disambiguation rivaling supervised methods. ACL’95. Eugene Agichtein Columbia University

Download ppt "Snowball : Extracting Relations from Large Plain-Text Collections"

Similar presentations

Cornell Accelerating Belief Propagation in Hardware Skand Hurkat and José Martínez Computer Systems Laboratory Cornell University

Cornell Accelerating Belief Propagation in Hardware Skand Hurkat and José Martínez Computer Systems Laboratory Cornell University
Distant Supervision for Relation Extraction without Labeled Data CSE 5539.

Distant Supervision for Relation Extraction without Labeled Data CSE 5539.
Data Mining Methodology 1. Why have a Methodology  Don’t want to learn things that aren’t true May not represent any underlying reality ○ Spurious correlation.

Data Mining Methodology 1. Why have a Methodology  Don’t want to learn things that aren’t true May not represent any underlying reality ○ Spurious correlation.
NYU ANLP-00 1 Automatic Discovery of Scenario-Level Patterns for Information Extraction Roman Yangarber Ralph Grishman Pasi Tapanainen Silja Huttunen.

NYU ANLP-00 1 Automatic Discovery of Scenario-Level Patterns for Information Extraction Roman Yangarber Ralph Grishman Pasi Tapanainen Silja Huttunen.
A Self Learning Universal Concept Spotter By Tomek Strzalkowski and Jin Wang Original slides by Iman Sen Edited by Ralph Grishman.

A Self Learning Universal Concept Spotter By Tomek Strzalkowski and Jin Wang Original slides by Iman Sen Edited by Ralph Grishman.
Multi-View Learning in the Presence of View Disagreement C. Mario Christoudias, Raquel Urtasun, Trevor Darrell UC Berkeley EECS & ICSI MIT CSAIL.

Multi-View Learning in the Presence of View Disagreement C. Mario Christoudias, Raquel Urtasun, Trevor Darrell UC Berkeley EECS & ICSI MIT CSAIL.
Query Dependent Pseudo-Relevance Feedback based on Wikipedia SIGIR ‘09 Advisor: Dr. Koh Jia-Ling Speaker: Lin, Yi-Jhen Date: 2010/01/24 1.

Query Dependent Pseudo-Relevance Feedback based on Wikipedia SIGIR ‘09 Advisor: Dr. Koh Jia-Ling Speaker: Lin, Yi-Jhen Date: 2010/01/24 1.
Automatic Classification of Text Databases Through Query Probing Panagiotis G. Ipeirotis Luis Gravano Columbia University Mehran Sahami E.piphany Inc.

Automatic Classification of Text Databases Through Query Probing Panagiotis G. Ipeirotis Luis Gravano Columbia University Mehran Sahami E.piphany Inc.
Text Classification With Support Vector Machines

Text Classification With Support Vector Machines
Clustering… in General In vector space, clusters are vectors found within  of a cluster vector, with different techniques for determining the cluster.

Clustering… in General In vector space, clusters are vectors found within  of a cluster vector, with different techniques for determining the cluster.
Unsupervised Models for Named Entity Classification Michael Collins and Yoram Singer Yimeng Zhang March 1 st, 2007.

Unsupervised Models for Named Entity Classification Michael Collins and Yoram Singer Yimeng Zhang March 1 st, 2007.
An Overview of Text Mining Rebecca Hwa 4/25/2002 References M. Hearst, “Untangling Text Data Mining,” in the Proceedings of the 37 th Annual Meeting of.

An Overview of Text Mining Rebecca Hwa 4/25/2002 References M. Hearst, “Untangling Text Data Mining,” in the Proceedings of the 37 th Annual Meeting of.
Combining Labeled and Unlabeled Data for Multiclass Text Categorization Rayid Ghani Accenture Technology Labs.

Combining Labeled and Unlabeled Data for Multiclass Text Categorization Rayid Ghani Accenture Technology Labs.
A Self Learning Universal Concept Spotter By Tomek Strzalkowski and Jin Wang Presented by Iman Sen.

A Self Learning Universal Concept Spotter By Tomek Strzalkowski and Jin Wang Presented by Iman Sen.
Co-training LING 572 Fei Xia 02/21/06. Overview Proposed by Blum and Mitchell (1998) Important work: –(Nigam and Ghani, 2000) –(Goldman and Zhou, 2000)

Co-training LING 572 Fei Xia 02/21/06. Overview Proposed by Blum and Mitchell (1998) Important work: –(Nigam and Ghani, 2000) –(Goldman and Zhou, 2000)
CS345 Data Mining Mining the Web for Structured Data.

CS345 Data Mining Mining the Web for Structured Data.
1 Extracting Relations from Large Text Collections Eugene Agichtein, Eleazar Eskin and Luis Gravano Department of Computer Science Columbia University.

1 Extracting Relations from Large Text Collections Eugene Agichtein, Eleazar Eskin and Luis Gravano Department of Computer Science Columbia University.
Information Extraction with Unlabeled Data Rayid Ghani Joint work with: Rosie Jones (CMU) Tom Mitchell (CMU & WhizBang! Labs) Ellen Riloff (University.

Information Extraction with Unlabeled Data Rayid Ghani Joint work with: Rosie Jones (CMU) Tom Mitchell (CMU & WhizBang! Labs) Ellen Riloff (University.
1 Modeling Query-Based Access to Text Databases Eugene Agichtein Panagiotis Ipeirotis Luis Gravano Computer Science Department Columbia University.

1 Modeling Query-Based Access to Text Databases Eugene Agichtein Panagiotis Ipeirotis Luis Gravano Computer Science Department Columbia University.
ISMB 2003 presentation Extracting Synonymous Gene and Protein Terms from Biological Literature Hong Yu and Eugene Agichtein Dept. Computer Science, Columbia.

ISMB 2003 presentation Extracting Synonymous Gene and Protein Terms from Biological Literature Hong Yu and Eugene Agichtein Dept. Computer Science, Columbia.

Similar presentations

Ads by Google