Presentation is loading. Please wait.

Presentation is loading. Please wait.

Question Classification II

Published byLisa Isabella Spencer Modified over 6 years ago

Similar presentations

Presentation on theme: "Question Classification II"— Presentation transcript:

1 Question Classification II
Ling573 NLP Systems and Applications April 30, 2013

2 Roadmap Question classification variations: Question series
SVM classifiers Sequence classifiers Sense information improvements Question series

3 Question Classification with Support Vector Machines
Hacioglu & Ward 2003 Same taxonomy, training, test data as Li & Roth

4 Question Classification with Support Vector Machines
Hacioglu & Ward 200 Same taxonomy, training, test data as Li & Roth Approach: Shallow processing Simpler features Strong discriminative classifiers

5 Question Classification with Support Vector Machines
Hacioglu & Ward 2003 Same taxonomy, training, test data as Li & Roth Approach: Shallow processing Simpler features Strong discriminative classifiers

6 Features & Processing Contrast: (Li & Roth)
POS, chunk info; NE tagging; other sense info

7 Features & Processing Contrast: (Li & Roth) Preprocessing:
POS, chunk info; NE tagging; other sense info Preprocessing: Only letters, convert to lower case, stopped, stemmed

8 Features & Processing Contrast: (Li & Roth) Preprocessing: Terms:
POS, chunk info; NE tagging; other sense info Preprocessing: Only letters, convert to lower case, stopped, stemmed Terms: Most informative 2000 word N-grams Identifinder NE tags (7 or 9 tags)

9 Classification & Results
Employs support vector machines for classification Best results: Bi-gram, 7 NE classes

10 Classification & Results
Employs support vector machines for classification Best results: Bi-gram, 7 NE classes Better than Li & Roth w/POS+chunk, but no semantics

11 Classification & Results
Employs support vector machines for classification Best results: Bi-gram, 7 NE classes Better than Li & Roth w/POS+chunk, but no semantics Fewer NE categories better More categories, more errors

12 Enhanced Answer Type Inference … Using Sequential Models
Krishnan, Das, and Chakrabarti 2005 Improves QC with CRF extraction of ‘informer spans’

13 Enhanced Answer Type Inference … Using Sequential Models
Krishnan, Das, and Chakrabarti 2005 Improves QC with CRF extraction of ‘informer spans’ Intuition: Humans identify Atype from few tokens w/little syntax

14 Enhanced Answer Type Inference … Using Sequential Models
Krishnan, Das, and Chakrabarti 2005 Improves QC with CRF extraction of ‘informer spans’ Intuition: Humans identify Atype from few tokens w/little syntax Who wrote Hamlet?

15 Enhanced Answer Type Inference … Using Sequential Models
Krishnan, Das, and Chakrabarti 2005 Improves QC with CRF extraction of ‘informer spans’ Intuition: Humans identify Atype from few tokens w/little syntax Who wrote Hamlet?

16 Enhanced Answer Type Inference … Using Sequential Models
Krishnan, Das, and Chakrabarti 2005 Improves QC with CRF extraction of ‘informer spans’ Intuition: Humans identify Atype from few tokens w/little syntax Who wrote Hamlet? How many dogs pull a sled at Iditarod?

17 Enhanced Answer Type Inference … Using Sequential Models
Krishnan, Das, and Chakrabarti 2005 Improves QC with CRF extraction of ‘informer spans’ Intuition: Humans identify Atype from few tokens w/little syntax Who wrote Hamlet? How many dogs pull a sled at Iditarod?

18 Enhanced Answer Type Inference … Using Sequential Models
Krishnan, Das, and Chakrabarti 2005 Improves QC with CRF extraction of ‘informer spans’ Intuition: Humans identify Atype from few tokens w/little syntax Who wrote Hamlet? How many dogs pull a sled at Iditarod? How much does a rhino weigh?

19 Enhanced Answer Type Inference … Using Sequential Models
Krishnan, Das, and Chakrabarti 2005 Improves QC with CRF extraction of ‘informer spans’ Intuition: Humans identify Atype from few tokens w/little syntax Who wrote Hamlet? How many dogs pull a sled at Iditarod? How much does a rhino weigh?

20 Enhanced Answer Type Inference … Using Sequential Models
Krishnan, Das, and Chakrabarti 2005 Improves QC with CRF extraction of ‘informer spans’ Intuition: Humans identify Atype from few tokens w/little syntax Who wrote Hamlet? How many dogs pull a sled at Iditarod? How much does a rhino weigh? Single contiguous span of tokens

21 Enhanced Answer Type Inference … Using Sequential Models
Krishnan, Das, and Chakrabarti 2005 Improves QC with CRF extraction of ‘informer spans’ Intuition: Humans identify Atype from few tokens w/little syntax Who wrote Hamlet? How many dogs pull a sled at Iditarod? How much does a rhino weigh? Single contiguous span of tokens

22 Enhanced Answer Type Inference … Using Sequential Models
Krishnan, Das, and Chakrabarti 2005 Improves QC with CRF extraction of ‘informer spans’ Intuition: Humans identify Atype from few tokens w/little syntax Who wrote Hamlet? How many dogs pull a sled at Iditarod? How much does a rhino weigh? Single contiguous span of tokens Who is the CEO of IBM?

23 Informer Spans as Features
Sensitive to question structure What is Bill Clinton’s wife’s profession?

24 Informer Spans as Features
Sensitive to question structure What is Bill Clinton’s wife’s profession?

25 Informer Spans as Features
Sensitive to question structure What is Bill Clinton’s wife’s profession? Idea: Augment Q classifier word ngrams w/IS info

26 Informer Spans as Features
Sensitive to question structure What is Bill Clinton’s wife’s profession? Idea: Augment Q classifier word ngrams w/IS info Informer span features: IS ngrams

27 Informer Spans as Features
Sensitive to question structure What is Bill Clinton’s wife’s profession? Idea: Augment Q classifier word ngrams w/IS info Informer span features: IS ngrams Informer ngrams hypernyms: Generalize over words or compounds

28 Informer Spans as Features
Sensitive to question structure What is Bill Clinton’s wife’s profession? Idea: Augment Q classifier word ngrams w/IS info Informer span features: IS ngrams Informer ngrams hypernyms: Generalize over words or compounds WSD?

29 Informer Spans as Features
Sensitive to question structure What is Bill Clinton’s wife’s profession? Idea: Augment Q classifier word ngrams w/IS info Informer span features: IS ngrams Informer ngrams hypernyms: Generalize over words or compounds WSD? No

30 Effect of Informer Spans
Classifier: Linear SVM + multiclass

31 Effect of Informer Spans
Classifier: Linear SVM + multiclass Notable improvement for IS hypernyms

32 Effect of Informer Spans
Classifier: Linear SVM + multiclass Notable improvement for IS hypernyms Better than all hypernyms – filter sources of noise Biggest improvements for ‘what’, ‘which’ questions

33 Perfect vs CRF Informer Spans

34 Recognizing Informer Spans
Idea: contiguous spans, syntactically governed

35 Recognizing Informer Spans
Idea: contiguous spans, syntactically governed Use sequential learner w/syntactic information

36 Recognizing Informer Spans
Idea: contiguous spans, syntactically governed Use sequential learner w/syntactic information Tag spans with B(egin),I(nside),O(outside) Employ syntax to capture long range factors

37 Recognizing Informer Spans
Idea: contiguous spans, syntactically governed Use sequential learner w/syntactic information Tag spans with B(egin),I(nside),O(outside) Employ syntax to capture long range factors Matrix of features derived from parse tree

38 Recognizing Informer Spans
Idea: contiguous spans, syntactically governed Use sequential learner w/syntactic information Tag spans with B(egin),I(nside),O(outside) Employ syntax to capture long range factors Matrix of features derived from parse tree Cell:x[i,l], i is position, l is depth in parse tree, only 2 Values: Tag: POS, constituent label in the position Num: number of preceding chunks with same tag

39 Parser Output Parse

40 Parse Tabulation Encoding and table:

41 CRF Indicator Features
Cell: IsTag, IsNum: e.g. y4 = 1 and x[4,2].tag=NP Also, IsPrevTag, IsNextTag

42 CRF Indicator Features
Cell: IsTag, IsNum: e.g. y4 = 1 and x[4,2].tag=NP Also, IsPrevTag, IsNextTag Edge: IsEdge: (u,v) , yi-1=u and yi=v IsBegin, IsEnd

43 CRF Indicator Features
Cell: IsTag, IsNum: e.g. y4 = 1 and x[4,2].tag=NP Also, IsPrevTag, IsNextTag Edge: IsEdge: (u,v) , yi-1=u and yi=v IsBegin, IsEnd All features improve

44 CRF Indicator Features
Cell: IsTag, IsNum: e.g. y4 = 1 and x[4,2].tag=NP Also, IsPrevTag, IsNextTag Edge: IsEdge: (u,v) , yi-1=u and yi=v IsBegin, IsEnd All features improve Question accuracy: Oracle: 88%; CRF: 86.2%

45 Question Classification Using Headwords and Their Hypernyms
Huang, Thint, and Qin 2008 Questions: Why didn’t WordNet/Hypernym features help in L&R?

46 Question Classification Using Headwords and Their Hypernyms
Huang, Thint, and Qin 2008 Questions: Why didn’t WordNet/Hypernym features help in L&R? Best results in L&R - ~200,000 feats; ~700 active Can we do as well with fewer features?

47 Question Classification Using Headwords and Their Hypernyms
Huang, Thint, and Qin 2008 Questions: Why didn’t WordNet/Hypernym features help in L&R? Best results in L&R - ~200,000 feats; ~700 active Can we do as well with fewer features? Approach: Refine features:

48 Question Classification Using Headwords and Their Hypernyms
Huang, Thint, and Qin 2008 Questions: Why didn’t WordNet/Hypernym features help in L&R? Best results in L&R - ~200,000 feats; ~700 active Can we do as well with fewer features? Approach: Refine features: Restrict use of WordNet to headwords

49 Question Classification Using Headwords and Their Hypernyms
Huang, Thint, and Qin 2008 Questions: Why didn’t WordNet/Hypernym features help in L&R? Best results in L&R - ~200,000 feats; ~700 active Can we do as well with fewer features? Approach: Refine features: Restrict use of WordNet to headwords Employ WSD techniques SVM, MaxEnt classifiers

50 Head Word Features Head words: Chunks and spans can be noisy

51 Head Word Features Head words: Chunks and spans can be noisy
E.g. Bought a share in which baseball team?

52 Head Word Features Head words: Chunks and spans can be noisy
E.g. Bought a share in which baseball team? Type: HUM: group (not ENTY:sport) Head word is more specific

53 Head Word Features Head words: Chunks and spans can be noisy
E.g. Bought a share in which baseball team? Type: HUM: group (not ENTY:sport) Head word is more specific Employ rules over parse trees to extract head words

54 Head Word Features Head words: Chunks and spans can be noisy
E.g. Bought a share in which baseball team? Type: HUM: group (not ENTY:sport) Head word is more specific Employ rules over parse trees to extract head words Issue: vague heads E.g. What is the proper name for a female walrus? Head = ‘name’?

55 Head Word Features Head words: Chunks and spans can be noisy
E.g. Bought a share in which baseball team? Type: HUM: group (not ENTY:sport) Head word is more specific Employ rules over parse trees to extract head words Issue: vague heads E.g. What is the proper name for a female walrus? Head = ‘name’? Apply fix patterns to extract sub-head (e.g. walrus)

56 Head Word Features Head words: Chunks and spans can be noisy
E.g. Bought a share in which baseball team? Type: HUM: group (not ENTY:sport) Head word is more specific Employ rules over parse trees to extract head words Issue: vague heads E.g. What is the proper name for a female walrus? Head = ‘name’? Apply fix patterns to extract sub-head (e.g. walrus) Also, simple regexp for other feature type E.g. ‘what is’ cue to definition type

57 WordNet Features Hypernyms:
Enable generalization: dog->..->animal Can generate noise: also

58 WordNet Features Hypernyms:
Enable generalization: dog->..->animal Can generate noise: also dog ->…-> person

59 WordNet Features Hypernyms: Adding low noise hypernyms
Enable generalization: dog->..->animal Can generate noise: also dog ->…-> person Adding low noise hypernyms Which senses?

60 WordNet Features Hypernyms: Adding low noise hypernyms
Enable generalization: dog->..->animal Can generate noise: also dog ->…-> person Adding low noise hypernyms Which senses? Restrict to matching WordNet POS

61 WordNet Features Hypernyms: Adding low noise hypernyms
Enable generalization: dog->..->animal Can generate noise: also dog ->…-> person Adding low noise hypernyms Which senses? Restrict to matching WordNet POS Which word senses?

62 WordNet Features Hypernyms: Adding low noise hypernyms
Enable generalization: dog->..->animal Can generate noise: also dog ->…-> person Adding low noise hypernyms Which senses? Restrict to matching WordNet POS Which word senses? Use Lesk algorithm: overlap b/t question & WN gloss

63 WordNet Features Hypernyms: Adding low noise hypernyms
Enable generalization: dog->..->animal Can generate noise: also dog ->…-> person Adding low noise hypernyms Which senses? Restrict to matching WordNet POS Which word senses? Use Lesk algorithm: overlap b/t question & WN gloss How deep?

64 WordNet Features Hypernyms: Adding low noise hypernyms
Enable generalization: dog->..->animal Can generate noise: also dog ->…-> person Adding low noise hypernyms Which senses? Restrict to matching WordNet POS Which word senses? Use Lesk algorithm: overlap b/t question & WN gloss How deep? Based on validation set: 6

65 WordNet Features Hypernyms: Adding low noise hypernyms
Enable generalization: dog->..->animal Can generate noise: also dog ->…-> person Adding low noise hypernyms Which senses? Restrict to matching WordNet POS Which word senses? Use Lesk algorithm: overlap b/t question & WN gloss How deep? Based on validation set: 6 Q Type similarity: compute similarity b/t headword & type Use type as feature

66 Other Features Question wh-word:
What,which,who,where,when,how,why, and rest

67 Other Features Question wh-word: N-grams: uni-,bi-,tri-grams
What,which,who,where,when,how,why, and rest N-grams: uni-,bi-,tri-grams

68 Other Features Question wh-word: N-grams: uni-,bi-,tri-grams
What,which,who,where,when,how,why, and rest N-grams: uni-,bi-,tri-grams Word shape: Case features: all upper, all lower, mixed, all digit, other

69 Results Per feature-type results:

70 Results: Incremental Additive improvement:

71 Error Analysis Inherent ambiguity: What is mad cow disease?
ENT: disease or DESC:def

72 Error Analysis Inherent ambiguity: Inconsistent labeling:
What is mad cow disease? ENT: disease or DESC:def Inconsistent labeling: What is the population of Kansas? NUM: other What is the population of Arcadia, FL ?

73 Error Analysis Inherent ambiguity: Inconsistent labeling: Parser error
What is mad cow disease? ENT: disease or DESC:def Inconsistent labeling: What is the population of Kansas? NUM: other What is the population of Arcadia, FL ? NUM:count Parser error

74 Question Classification: Summary
Issue: Integrating rich features/deeper processing

75 Question Classification: Summary
Issue: Integrating rich features/deeper processing Errors in processing introduce noise

76 Question Classification: Summary
Issue: Integrating rich features/deeper processing Errors in processing introduce noise Noise in added features increases error

77 Question Classification: Summary
Issue: Integrating rich features/deeper processing Errors in processing introduce noise Noise in added features increases error Large numbers of features can be problematic for training

78 Question Classification: Summary
Issue: Integrating rich features/deeper processing Errors in processing introduce noise Noise in added features increases error Large numbers of features can be problematic for training Alternative solutions:

79 Question Classification: Summary
Issue: Integrating rich features/deeper processing Errors in processing introduce noise Noise in added features increases error Large numbers of features can be problematic for training Alternative solutions: Use more accurate shallow processing, better classifier

80 Question Classification: Summary
Issue: Integrating rich features/deeper processing Errors in processing introduce noise Noise in added features increases error Large numbers of features can be problematic for training Alternative solutions: Use more accurate shallow processing, better classifier Restrict addition of features to

81 Question Classification: Summary
Issue: Integrating rich features/deeper processing Errors in processing introduce noise Noise in added features increases error Large numbers of features can be problematic for training Alternative solutions: Use more accurate shallow processing, better classifier Restrict addition of features to Informer spans Headwords

82 Question Classification: Summary
Issue: Integrating rich features/deeper processing Errors in processing introduce noise Noise in added features increases error Large numbers of features can be problematic for training Alternative solutions: Use more accurate shallow processing, better classifier Restrict addition of features to Informer spans Headwords Filter features to be added

Download ppt "Question Classification II"

Similar presentations

Sequence Classification: Chunking Shallow Processing Techniques for NLP Ling570 November 28, 2011.

Sequence Classification: Chunking Shallow Processing Techniques for NLP Ling570 November 28, 2011.
CSCI 347 / CS 4206: Data Mining Module 07: Implementations Topic 03: Linear Models.

CSCI 347 / CS 4206: Data Mining Module 07: Implementations Topic 03: Linear Models.
Max-Margin Matching for Semantic Role Labeling David Vickrey James Connor Daphne Koller Stanford University.

Max-Margin Matching for Semantic Role Labeling David Vickrey James Connor Daphne Koller Stanford University.
Made with OpenOffice.org 1 Sentiment Classification using Word Sub-Sequences and Dependency Sub-Trees Pacific-Asia Knowledge Discovery and Data Mining.

Made with OpenOffice.org 1 Sentiment Classification using Word Sub-Sequences and Dependency Sub-Trees Pacific-Asia Knowledge Discovery and Data Mining.
Shallow Parsing CS 4705 Julia Hirschberg 1. Shallow or Partial Parsing Sometimes we don’t need a complete parse tree –Information extraction –Question.

Shallow Parsing CS 4705 Julia Hirschberg 1. Shallow or Partial Parsing Sometimes we don’t need a complete parse tree –Information extraction –Question.
Question Classification (cont’d) Ling573 NLP Systems & Applications April 12, 2011.

Question Classification (cont’d) Ling573 NLP Systems & Applications April 12, 2011.
Jiho Han Ronny (Dowon) Ko.  Objective: automatically generate the summary of review extracting the strength/weakness of the product  Use NLP techniques.

Jiho Han Ronny (Dowon) Ko.  Objective: automatically generate the summary of review extracting the strength/weakness of the product  Use NLP techniques.
Shallow Processing: Summary Shallow Processing Techniques for NLP Ling570 December 7, 2011.

Shallow Processing: Summary Shallow Processing Techniques for NLP Ling570 December 7, 2011.
CS 47051 Word Sense Disambiguation. 2 Overview A problem for semantic attachment approaches: what happens when a given lexeme has multiple ‘meanings’?

CS Word Sense Disambiguation. 2 Overview A problem for semantic attachment approaches: what happens when a given lexeme has multiple ‘meanings’?
Introduction to CL Session 1: 7/08/2011. What is computational linguistics? Processing natural language text by computers  for practical applications.

Introduction to CL Session 1: 7/08/2011. What is computational linguistics? Processing natural language text by computers  for practical applications.
Feature Screening Concept: A greedy feature selection method. Rank features and discard those whose ranking criterions are below the threshold. Problem:

Feature Screening Concept: A greedy feature selection method. Rank features and discard those whose ranking criterions are below the threshold. Problem:
Course Summary LING 572 Fei Xia 03/06/07. Outline Problem description General approach ML algorithms Important concepts Assignments What’s next?

Course Summary LING 572 Fei Xia 03/06/07. Outline Problem description General approach ML algorithms Important concepts Assignments What’s next?
The classification problem (Recap from LING570) LING 572 Fei Xia, Dan Jinguji Week 1: 1/10/08 1.

The classification problem (Recap from LING570) LING 572 Fei Xia, Dan Jinguji Week 1: 1/10/08 1.
STRUCTURED PERCEPTRON Alice Lai and Shi Zhi. Presentation Outline Introduction to Structured Perceptron ILP-CRF Model Averaged Perceptron Latent Variable.

STRUCTURED PERCEPTRON Alice Lai and Shi Zhi. Presentation Outline Introduction to Structured Perceptron ILP-CRF Model Averaged Perceptron Latent Variable.
Mining the Peanut Gallery: Opinion Extraction and Semantic Classification of Product Reviews K. Dave et al, WWW 2003, 1480+ citations Presented by Sarah.

Mining the Peanut Gallery: Opinion Extraction and Semantic Classification of Product Reviews K. Dave et al, WWW 2003, citations Presented by Sarah.
COMP423: Intelligent Agent Text Representation. Menu – Bag of words – Phrase – Semantics – Bag of concepts – Semantic distance between two words.

COMP423: Intelligent Agent Text Representation. Menu – Bag of words – Phrase – Semantics – Bag of concepts – Semantic distance between two words.
Authors: Ting Wang, Yaoyong Li, Kalina Bontcheva, Hamish Cunningham, Ji Wang Presented by: Khalifeh Al-Jadda Automatic Extraction of Hierarchical Relations.

Authors: Ting Wang, Yaoyong Li, Kalina Bontcheva, Hamish Cunningham, Ji Wang Presented by: Khalifeh Al-Jadda Automatic Extraction of Hierarchical Relations.
Chapter 10: Compilers and Language Translation Invitation to Computer Science, Java Version, Third Edition.

Chapter 10: Compilers and Language Translation Invitation to Computer Science, Java Version, Third Edition.
Ling 570 Day 17: Named Entity Recognition Chunking.

Ling 570 Day 17: Named Entity Recognition Chunking.
Complex Linguistic Features for Text Classification: A Comprehensive Study Alessandro Moschitti and Roberto Basili University of Texas at Dallas, University.

Complex Linguistic Features for Text Classification: A Comprehensive Study Alessandro Moschitti and Roberto Basili University of Texas at Dallas, University.

Similar presentations

Ads by Google