Presentation is loading. Please wait.

Presentation is loading. Please wait.

CSC 594 Topics in AI – Natural Language Processing

Similar presentations


Presentation on theme: "CSC 594 Topics in AI – Natural Language Processing"— Presentation transcript:

1 CSC 594 Topics in AI – Natural Language Processing
Spring 2016/17 4. Text Normalization (Some slides adapted from Jurafsky & Martin)

2 Text Normalization Every NLP task needs to do some form of text normalization. Not terribly glamorous but critical to further processing Determining a relevant vocabulary Segmenting/tokenizing words in running text Normalizing word formats Segmenting sentences in running text

3 Speech and Language Processing - Jurafsky and Martin
How many words? “They lay back on the San Francisco grass and looked at the stars and their…” Type: an element of the vocabulary Token: an instance of that type in running text How many words are there in this sentence? 15 tokens (or maybe 14) 13 types (or 12) Speech and Language Processing - Jurafsky and Martin

4 Speech and Language Processing - Jurafsky and Martin
Vocabulary First, what’s our vocabulary? What words are needed for a particular application? All of English? A subset? How many will there be? How do we determine the right list? Is it finite? Etc. Speech and Language Processing - Jurafsky and Martin

5 Speech and Language Processing - Jurafsky and Martin
How many words? N = number of tokens V = vocabulary = set of types |V| is the size of the vocabulary Tokens = N Types = |V| Switchboard phone conversations 2.4 million 20 thousand Shakespeare 884,000 31 thousand Google N-grams 1 trillion 13 million Speech and Language Processing - Jurafsky and Martin

6 Simple Tokenization in UNIX
Given a text file, output all the word types and their associated frequencies in that text corpus Inspired by Ken Church’s UNIX for Poets. Unix has many commands to deal with basic text processing operations Original Unix designers cared a lot about text processing Most of which are line oriented Speech and Language Processing - Jurafsky and Martin

7 The first step: tokenizing
So let’s do some dumb tokenization and get each token on a line by itself. tr -sc ’A-Za-z’ ’\n’ < shakes.txt | head THE SONNETS by William Shakespeare From fairest creatures We ... Speech and Language Processing - Jurafsky and Martin

8 The second step: sorting
tr -sc ’A-Za-z’ ’\n’ < shakes.txt | sort | head A ... Speech and Language Processing - Jurafsky and Martin

9 Speech and Language Processing - Jurafsky and Martin
Then some counting Merging upper and lower case tr ‘A-Z’ ‘a-z’ < shakes.txt | tr –sc ‘A-Za-z’ ‘\n’ Sorting the counts tr ‘A-Z’ ‘a-z’ < shakes.txt | tr –sc ‘A-Za-z’ ‘\n’ | sort | uniq –c | sort –n –r 23243 the 22225 i 18618 and 16339 to 15687 of 12780 a 12163 you 10839 my 10005 in 8954 d What happened here? Speech and Language Processing - Jurafsky and Martin

10 Issues in Tokenization
Finland’s capital  Finland Finlands Finland’s ? what’re, I’m, isn’t  What are, I am, is not Hewlett-Packard  Hewlett Packard ? state-of-the-art  state of the art ? Lowercase  lower-case lowercase lower case ? San Francisco  one token or two? m.p.g., PhD.  ?? Speech and Language Processing - Jurafsky and Martin

11 Tokenization: language issues
French L'ensemble  one token or two? L ? L’ ? Le ? German noun compounds are not segmented Lebensversicherungsgesellschaftsangestellter ‘life insurance company employee’ German information retrieval needs compound splitter Speech and Language Processing - Jurafsky and Martin

12 Tokenization: language issues
Chinese has no spaces between words 莎拉波娃现在居住在美国东南部的佛罗里达。 莎拉波娃 现在 居住 在 美国 东南部 的 佛罗里达 Sharapova now lives in US southeastern Florida Japanese allows multiple alphabets intermingled フォーチュン500社は情報不足のため時間あた$500K(約6,000万円) Katakana Hiragana Kanji Romaji Speech and Language Processing - Jurafsky and Martin

13 Word Tokenization in Chinese
Also called Word Segmentation Chinese words are composed of characters Characters are generally 1 syllable and 1 morpheme. Average word is 2.4 characters long. Standard baseline segmentation algorithm: Maximum Matching (also called Greedy) Speech and Language Processing - Jurafsky and Martin

14 Maximum Matching Word Segmentation Algorithm
Given a wordlist of Chinese, and a string Start a pointer at the beginning of the string Find the longest word in dictionary that matches the string starting at pointer Move the pointer over the word in string Go to 2 Speech and Language Processing - Jurafsky and Martin

15 Speech and Language Processing - Jurafsky and Martin

16 Max-match English Example
Thecatinthehat Thetabledownthere Doesn’t generally work in English! But works astonishingly well in Chinese 莎拉波娃现在居住在美国东南部的佛罗里达。 莎拉波娃 现在 居住 在 美国 东南部 的 佛罗里达 Modern probabilistic segmentation algorithms even better the cat in the hat the table down there theta bled own there Speech and Language Processing - Jurafsky and Martin

17 Speech and Language Processing - Jurafsky and Martin
Case folding Applications like web search: reduce all letters to lower case Since users tend to use lower case For sentiment analysis, MT, Information extraction Case is helpful (US versus us; IRE vs. ire) Speech and Language Processing - Jurafsky and Martin

18 Speech and Language Processing - Jurafsky and Martin
Lemmatization Reduce inflections or variant forms to base form am, are, is  be car, cars, car's, cars'  car the boy's cars are different colors  the boy car be different color Lemmatization: have to find correct dictionary headword form Speech and Language Processing - Jurafsky and Martin

19 Speech and Language Processing - Jurafsky and Martin
Stemming Reduce terms to their stems Stemming is crude chopping of affixes language dependent e.g., automate(s), automatic, automation all reduced to automat. for example compressed and compression are both accepted as equivalent to compress. for exampl compress and compress ar both accept as equival to compress Speech and Language Processing - Jurafsky and Martin

20 Speech and Language Processing - Jurafsky and Martin
Porter’s Algorithm Step 1a sses  ss caresses  caress ies  i ponies  poni ss  ss caress  caress s  ø cats  cat Step 1b (*v*)ing  ø walking  walk sing  sing (*v*)ed  ø plastered  plaster Step 2 (for long stems) ational ate relational relate izer ize digitizer  digitize ator ate operator  operate Step 3 (for longer stems) al  ø revival  reviv able  ø adjustable  adjust ate  ø activate  activ Speech and Language Processing - Jurafsky and Martin

21 Speech and Language Processing - Jurafsky and Martin
Complex Morphology Some languages requires complex morpheme segmentation Turkish Uygarlastiramadiklarimizdanmissinizcasina `(behaving) as if you are among those whom we could not civilize’ Uygar `civilized’ + las `become’ + tir `cause’ + ama `not able’ + dik `past’ + lar ‘plural’ + imiz ‘p1pl’ + dan ‘abl’ + mis ‘past’ + siniz ‘2pl’ + casina ‘as if’ Speech and Language Processing - Jurafsky and Martin

22 Sentence Segmentation
In English, punctuation is used to mark sentence boundaries !, ? are relatively unambiguous Period “.” is quite ambiguous Abbreviations like Inc. or Dr. Numbers like .02% or 4.3 Machine learning approach Build a binary classifier Looks at each possible EOS punctuation and decides EndOfSentence/NotEndOfSentence Speech and Language Processing - Jurafsky and Martin

23 Speech and Language Processing - Jurafsky and Martin
Decision Tree Example Speech and Language Processing - Jurafsky and Martin

24 More sophisticated decision tree features
Case of word with “.”: Upper, Lower, Cap, Number Case of word after “.”: Upper, Lower, Cap, Number Numeric features Length of word with “.” Probability(word with “.” occurs at end-of-s) Probability(word after “.” occurs at beginning-of-s) Speech and Language Processing - Jurafsky and Martin

25 Decision Trees and other classifiers
We can think of the questions in a decision tree as features that could be exploited by any kind of classifier Logistic regression SVM Neural Nets etc. Speech and Language Processing - Jurafsky and Martin

26 Speech and Language Processing - Jurafsky and Martin
Maximum Matching themartian Speech and Language Processing - Jurafsky and Martin

27 Testing, Improvement and Evaluation
There are two tasks: Make sure that you have implemented MaxMatch correctly Testing Figure out a way to improve performance on this task This means you need a way to detect improvement Speech and Language Processing - Jurafsky and Martin

28 Speech and Language Processing - Jurafsky and Martin
Testing Given a particular dictionary and a correct implementation there is a “right” answer that MaxMatch should come up with. So for “themartian” that might be them art i an That’s the “right” answer even though its clearly not the right answer… Speech and Language Processing - Jurafsky and Martin

29 Improvement and Evaluation
So given a test set of outputs like them art i an How do we know if things are getting better? What do we need to know to say things are getting better? Speech and Language Processing - Jurafsky and Martin

30 Speech and Language Processing - Jurafsky and Martin
Reference Answers We need to know what the “right” answer is. Here “right” means the answer we would expect a human to produce. In this case “the martian”. So we have them art i an the martian And a whole bunch of examples like this, some right, some wrong How do we assess how well we’re doing? Speech and Language Processing - Jurafsky and Martin

31 Speech and Language Processing - Jurafsky and Martin
Evaluation Strict accuracy Given a test set, how many things are right and how many are wrong? Too pessimistic Might get you fired Not fine-grained enough May not show you are making progress when you are in fact making progress Speech and Language Processing - Jurafsky and Martin

32 Speech and Language Processing - Jurafsky and Martin
Progress? Start with them art I an the martian Move to The mart I an The martian They’re both still wrong. So does it make sense to say one is better? Speech and Language Processing - Jurafsky and Martin


Download ppt "CSC 594 Topics in AI – Natural Language Processing"

Similar presentations


Ads by Google