Presentation is loading. Please wait.

Presentation is loading. Please wait.

Outline POS tagging Tag wise accuracy Graph- tag wise accuracy

Published byRoxanne Atkins Modified over 9 years ago

Similar presentations

Presentation on theme: "Outline POS tagging Tag wise accuracy Graph- tag wise accuracy"— Presentation transcript:

1 Natural Language Processing Assignment Group Members: Soumyajit De Naveen Bansal Sanobar Nishat

2 Outline POS tagging Tag wise accuracy Graph- tag wise accuracy
Precision recall f-score Improvements In POS tagging Implementation of tri-gram POS tagging with smoothing Improved precision, recall and f-score Comparison between Discriminative and Generative Model Next word prediction Model #1 Model #2 Implementation method and details Scoring ratio perplexity ratio

3 Outline NLTK Yago Different examples by using Yago Parsing
Conclusions A* Implementation – A Comparison with Viterbi

4 Assignment#1 HMM Based POS Tagger

5 Tag Wise Accuracy

6 Graph – Tag Wise Accuracy

7 Precision, Recall, F-Score
Precision : tp/(tp+fp) = 0.92 Recall: tp/(tp+fn) = 1 F-score: 2.precision.recall/(precision + recall) = 0.958

8 Improvements in HMM Based POS Tagger
Assignment#1 (Cont..) Improvements in HMM Based POS Tagger

9 Improvement in HMM Based POS Tagger
Implementation of Trigram * Issues - sparcity * Solution – Implementation of smoothing techniques * Results – increases overall accuracy up to 94%

10 Smoothing Technique Implementation of Smoothing Technique
* Linear Interpolation Technique * Formula: i.e. * Finding value of lambda (discussed in “ TnT- A Statistical Part-of-Speech Tagger ”)

11 POS Tagging Accuracy With Smoothing

12 Precision, Recall, F-Score
Precision : tp/(tp+fp) = Recall: tp/(tp+fn) = 1 F-score: 2.precision.recall/(precision + recall) = 0.97

13 Tag Wise Accuracy

14 Tag wise accuracy (cont..)

15 Improvements in HMM Based POS Tagger Handling Unknown Words
Assignment#1 (Cont..) Improvements in HMM Based POS Tagger Handling Unknown Words

16 Precision Score (accuracy in Percentage)

17 Tag Wise Accuracy

18 Error Analysis (Tag Wise Accuracy)
VVB - finite base form of lexical verbs (e.g. forget, send, live, return) Count: 9916 Confused with counts Reason VVI (infinitive form of lexical verbs (e.g. forget, send, live, return)) 1201 VVB is used to tagged the word that has the same form as the infinitive without “to” for all persons. E.g. He has to show Show me VVD (The past tense form of lexical verbs (e.g. forgot, sent, lived, returned)) 145 The base form and past tense form of many verbs are same. So domination of emission probability of such word caused VVB wrongly tagged as VVD. And effect of transition probability might got have lower influence. NN1 303 Words with similar base form gets confuse with common noun. e.g. The seasonally adjusted total regarded as… Total has been tagged as VVB and NN1

19 Error Analysis (cont..) ZZ0 - Alphabetical symbols (e.g. A, a, B, b, c, d) (Accuracy - 63%) Count: 337 Confused with counts Reason AT0 (Article e.g. the, a, an, no) 98 Emission probability of “a” as AT0 is much higher compare to ZZ0. Hence AT0 dominates while tagging “a” CRD (Cardinal number e.g. one, 3, fifty-five, 3609) 16 Because of the assumption of bigram/trigram Transition probability.

20 Error Analysis (cont..) ITJ - Interjection (Accuracy - 65%) Count: 177
Reason: ITJ Tag appeared so less number of times, that it didn't miss classified that much, but yet its percentage is so low Confused with counts Reason AT0 (Article (e.g. the, a, an, no)) 26 “No“ is used as ITJ and article in the corpus. So confusion is due to the higher emission probability of word with AT0 NN1 (Singular common noun) 14 “Bravo” is tagged as NN1 and ITJ in corpus

21 Error Analysis (cont..) UNC - Unclassified items (Accuracy - 23%)
Count: 756 Confused with counts Reason AT0 (Article (e.g. the, a, an, no)) 69 Because of the domination of transition probability UNC is wrongly tagged NN1 (Singular common noun) 224 NP0 (Proper noun (e.g. London, Michael, Mars, IBM)) 132 New word with begin capital letter is tagged as NP0, since mostly the UNC words are not repeating among different corpus.

22 Assignment#2 Discriminative & Generative Model – A Comparison

23 Discriminative and Generative Model

24 Comparison Graph

25 Conclusion Since its unigram, Discriminative and Generative Model are giving same performance, as expected

26 Assignment#3 Next word prediction

27 Model # 1 When only previous word is given Example: He likes

28 Model # 2 When previous Tag & previous word are known. Example: He_PP0 likes_VB Previous Work

29 Model # 2 (cont..) Current Work

30 Evaluation Method 1. Scoring Method
Divide the testing corpus into bigram Match the testing corpus 2nd word of bigram with predicted word of each model Increment the score if match found The final evaluation is the ratio of the two scores of each model i.e. model1/model2 If ratio > 1 => model 1 is performing better and vice-verca.

31 Implementation Detail
Look Up Table Previous Word Next Predicted Word (Model 1) Next Predicted Word (Model 2) I see he looks goes : Look up is used in predicting the next word

32 Scoring Ratio

33 2. Perplexity: Comparison:

34 Perplexity Ratio

35 Remarks Model 2 is performing poorer than model 1 because of words are sparse among tags.

36 Next word prediction Further Experiments
Assignment#3 (Cont..) Next word prediction Further Experiments

37 Score (ratio) of word-prediction

38 Perplexity (ratio) of word-prediction

39 Remarks Perplexity is found to be decreasing in this model.
Overall score has been increased.

40 Assignment#4 Yaqo

41 Example #1 Query : Amitabh and Sachin
wikicategory_Living_people -- <type> -- Amitabh_Bachchan -- <givenNameOf> -- Amitabh wikicategory_Living_people -- <type> -- Sachin_Tendulkar -- <givenNameOf> -- Sachin ANOTHER-PATH wikicategory_Padma_Shri_recipients -- <type> -- Amitabh_Bachchan -- <givenNameOf> -- Amitabh wikicategory_Padma_Shri_recipients -- <type> -- Sachin_Tendulkar -- <givenNameOf> -- Sachin

42 Example#2 Query : India and Pakistan PATH
wikicategory_WTO_member_economies -- <type> -- India wikicategory_WTO_member_economies -- <type> -- Pakistan ANOTHER-PATH wikicategory_English-speaking_countries_and_territories -- <type> -- India wikicategory_English-speaking_countries_and_territories -- <type> -- Pakistan Operation_Meghdoot -- <participatedIn> -- India Operation_Meghdoot -- <participatedIn> -- Pakistan

43 ANOTHER-PATH Operation_Trident_(Indo-Pakistani_War) -- <participatedIn> -- India Operation_Trident_(Indo-Pakistani_War) -- <participatedIn> -- Pakistan Siachen_conflict -- <participatedIn> -- India Siachen_conflict -- <participatedIn> -- Pakistan wikicategory_Asian_countries -- <type> -- India wikicategory_Asian_countries -- <type> -- Pakistan

44 ANOTHER-PATH Capture_of_Kishangarh_Fort -- <participatedIn> -- India Capture_of_Kishangarh_Fort -- <participatedIn> -- Pakistan wikicategory_South_Asian_countries -- <type> -- India wikicategory_South_Asian_countries -- <type> -- Pakistan Operation_Enduring_Freedom -- <participatedIn> -- India Operation_Enduring_Freedom -- <participatedIn> -- Pakistan wordnet_region_ <type> -- India wordnet_region_ <type> -- Pakistan

45 Example #3 Query: Tom and Jerry
wikicategory_Living_people -- <type> -- Tom_Green -- <givenNameOf> -- Tom wikicategory_Living_people -- <type> -- Jerry_Brown -- <givenNameOf> -- Jerry

46 Assignment#5 Parser projection

47 Example#1

48 Example#2

49 Example#3

50 Example#5 Example#4

51 Example#6

52 Example#7

53 Example#8

54 Conclusion 1. VBZ always comes at the end of the parse tree in Hindi and Urdu. 2. The structure in Hindi and Urdu is always expand or reset to NP VB e.g. S=> NP VP (no change) OR VP => VBZ NP (interchange) 3. For exact translation in Hindi and Urdu, merging of sub-tree in english is sometimes required 4. One word to multiple words mapping is common while translating from English to Hindi/Urdu e.g. donar => aatiya shuda OR have => rakhta hai 5. Phrase to phrase translation is sometimes required, so chunking is required e.g. hand in hand => choli daman ka saath (Urdu) => sath sath hain (Hindi) 6. DT NN or DT NP doesn’t interchange 7. In example#7: correct translation won’t require merging of two sub-trees MD and VP e.g. could be => jasakta hai

55 NLTK Toolkit NLTK is a suite of open source Python modules
Components of NLTK : Code, Corpora >30 annotated data sets corpus readers tokenizers stemmers taggers parsers wordnet semantic interpretation

56 Assignment#6 A* Implementation & Comparison with Vitervi

57 ^ A* - Heuristic $ A B C D Selected Route Transition probability
Fixed cost at each level (L)= (Min cost)* No. of Hops

58 Heuristic(h) F(B) = g(B) + h(B)
where h(B) = min(i,j){-log(Pr(Cj|Bi) * Pr(Wc|Cj))} + min(i,j){-log(Pr(tj|ti) * Pr(Wtj|tj))} * (n-2) + min(k){-log(Pr($|Dk) * Pr($|$))} Here, n = #nodes from Bi to $ (including $) Wc = word emitted from next node, C ti ,tj = any combination of tags in the graph Wtj = word emitted from the node tj

59 Result Viterbi / A* Ratio : score(Viterbi) / score(A*) = 1.0 Where,
score(Algo) = #correct predictions in the test corpus

60 Conclusion Since we are making bigram assumption and Viterbi is pruned in a careful way that is guaranteed to find the optimal path in a bigram HMM, its giving the optimal path. For A*, since our heuristic is underestimating and also maintains triangular inequality, A* is also giving the optimal path in the graph. Since A* has to backtrack at times, it requires more time and memory to find the solution compared to Viterbi.

Download ppt "Outline POS tagging Tag wise accuracy Graph- tag wise accuracy"

Similar presentations

Three Basic Problems Compute the probability of a text: P m (W 1,N ) Compute maximum probability tag sequence: arg max T 1,N P m (T 1,N | W 1,N ) Compute.

Three Basic Problems Compute the probability of a text: P m (W 1,N ) Compute maximum probability tag sequence: arg max T 1,N P m (T 1,N | W 1,N ) Compute.
School of something FACULTY OF OTHER School of Computing FACULTY OF ENGINEERING Machine Learning PoS-Taggers COMP3310 Natural Language Processing Eric.

School of something FACULTY OF OTHER School of Computing FACULTY OF ENGINEERING Machine Learning PoS-Taggers COMP3310 Natural Language Processing Eric.
School of something FACULTY OF OTHER School of Computing FACULTY OF ENGINEERING PoS-Tagging theory and terminology COMP3310 Natural Language Processing.

School of something FACULTY OF OTHER School of Computing FACULTY OF ENGINEERING PoS-Tagging theory and terminology COMP3310 Natural Language Processing.
Three Basic Problems 1.Compute the probability of a text (observation) language modeling – evaluate alternative texts and models P m (W 1,N ) 2.Compute.

Three Basic Problems 1.Compute the probability of a text (observation) language modeling – evaluate alternative texts and models P m (W 1,N ) 2.Compute.
CS460/IT632 Natural Language Processing/Language Technology for the Web Lecture 2 (06/01/06) Prof. Pushpak Bhattacharyya IIT Bombay Part of Speech (PoS)

CS460/IT632 Natural Language Processing/Language Technology for the Web Lecture 2 (06/01/06) Prof. Pushpak Bhattacharyya IIT Bombay Part of Speech (PoS)
Ling 570 Day 6: HMM POS Taggers 1. Overview Open Questions HMM POS Tagging Review Viterbi algorithm Training and Smoothing HMM Implementation Details.

Ling 570 Day 6: HMM POS Taggers 1. Overview Open Questions HMM POS Tagging Review Viterbi algorithm Training and Smoothing HMM Implementation Details.
CS626 Assignments Abhirut Gupta Mandar Joshi Piyush Dungarwal (Group No. 6)

CS626 Assignments Abhirut Gupta Mandar Joshi Piyush Dungarwal (Group No. 6)
Final Assignment Demo 11 th Nov, 2012 Deepak Suyel Geetanjali Rakshit Sachin Pawar CS 626 – Sppech, NLP and the Web.

Final Assignment Demo 11 th Nov, 2012 Deepak Suyel Geetanjali Rakshit Sachin Pawar CS 626 – Sppech, NLP and the Web.
Parts of Speech Generally speaking, the “grammatical type” of word: –Verb, Noun, Adjective, Adverb, Article, … We can also include inflection: –Verbs:

Parts of Speech Generally speaking, the “grammatical type” of word: –Verb, Noun, Adjective, Adverb, Article, … We can also include inflection: –Verbs:
Part of Speech Tagging The DT students NN went VB to P class NN Plays VB NN well ADV NN with P others NN DT Fruit NN flies NN VB NN VB like VB P VB a DT.

Part of Speech Tagging The DT students NN went VB to P class NN Plays VB NN well ADV NN with P others NN DT Fruit NN flies NN VB NN VB like VB P VB a DT.
Learning with Probabilistic Features for Improved Pipeline Models Razvan C. Bunescu Electrical Engineering and Computer Science Ohio University Athens,

Learning with Probabilistic Features for Improved Pipeline Models Razvan C. Bunescu Electrical Engineering and Computer Science Ohio University Athens,
In Search of a More Probable Parse: Experiments with DOP* and the Penn Chinese Treebank Aaron Meyers Linguistics 490 Winter 2009.

In Search of a More Probable Parse: Experiments with DOP* and the Penn Chinese Treebank Aaron Meyers Linguistics 490 Winter 2009.
Hidden Markov Model (HMM) Tagging  Using an HMM to do POS tagging  HMM is a special case of Bayesian inference.

Hidden Markov Model (HMM) Tagging  Using an HMM to do POS tagging  HMM is a special case of Bayesian inference.
Tagging with Hidden Markov Models. Viterbi Algorithm. Forward-backward algorithm Reading: Chap 6, Jurafsky & Martin Instructor: Paul Tarau, based on Rada.

Tagging with Hidden Markov Models. Viterbi Algorithm. Forward-backward algorithm Reading: Chap 6, Jurafsky & Martin Instructor: Paul Tarau, based on Rada.
Part II. Statistical NLP Advanced Artificial Intelligence Part of Speech Tagging Wolfram Burgard, Luc De Raedt, Bernhard Nebel, Lars Schmidt-Thieme Most.

Part II. Statistical NLP Advanced Artificial Intelligence Part of Speech Tagging Wolfram Burgard, Luc De Raedt, Bernhard Nebel, Lars Schmidt-Thieme Most.
PCFG Parsing, Evaluation, & Improvements Ling 571 Deep Processing Techniques for NLP January 24, 2011.

PCFG Parsing, Evaluation, & Improvements Ling 571 Deep Processing Techniques for NLP January 24, 2011.
Part-of-speech Tagging cs224n Final project Spring, 2008 Tim Lai.

Part-of-speech Tagging cs224n Final project Spring, 2008 Tim Lai.
Ch 10 Part-of-Speech Tagging Edited from: L. Venkata Subramaniam February 28, 2002.

Ch 10 Part-of-Speech Tagging Edited from: L. Venkata Subramaniam February 28, 2002.
Predicting the Semantic Orientation of Adjective Vasileios Hatzivassiloglou and Kathleen R. McKeown Presented By Yash Satsangi.

Predicting the Semantic Orientation of Adjective Vasileios Hatzivassiloglou and Kathleen R. McKeown Presented By Yash Satsangi.
Statistical Phrase-Based Translation Authors: Koehn, Och, Marcu Presented by Albert Bertram Titles, charts, graphs, figures and tables were extracted from.

Statistical Phrase-Based Translation Authors: Koehn, Och, Marcu Presented by Albert Bertram Titles, charts, graphs, figures and tables were extracted from.

Similar presentations

Ads by Google