1. LING 388: Computers and Language
Lecture 23

2. nltk book: chapter 3
Last time:

3. nltk book: chapter 3 Download: raw file = 1 (long) string
Text number 2554 is an English translation of Crime and Punishment

4. nltk book: chapter 3
Adjusting start and end:

5. nltk book: chapter 3 Searching Tokenized Text in nltk
angle brackets <…> mark token boundaries
>>> text[:20]
['Title', ':', 'Mrs.', 'Dalloway', 'Author', ':', 'Virginia', 'Woolf', 'Mrs.', 'Dalloway', 'said', 'she', 'would', 'buy', 'the', 'flowers', 'herself', '.', 'For', 'Lucy']
>>> text.findall(r"<Mrs\.> (<\w+>)")
Dalloway; Dalloway; Foxcroft; Dalloway; Asquith; Dalloway; Richard;
Dalloway; Dalloway; Dalloway; Coates; Coates; Bletchley; Bletchley;
Dempster; Dempster; Dempster; Dempster; Dempster; Dempster; Dempster;
Dalloway; Walker; Dalloway; Walker; Dalloway; Dalloway; Dalloway;
Dalloway; Turner; Filmer; Hugh; Septimus; Filmer; Filmer; Warren;
Smith; Filmer; Smith; Warren; Dalloway; Whitbread; Marsham; Marsham;
Marsham; Marsham; Hilbery; Dalloway; Dalloway; Dalloway; Dalloway;
Dalloway; Dalloway; Marsham; Marsham; Dalloway; Dalloway; Gorham;
Dalloway; Filmer; Peters; Peters; Filmer; Peters; Peters; Filmer;
Peters; Peters; Peters; Peters; Filmer; Peters; Peters; Peters;
Filmer; Filmer; Filmer; Williams; Filmer; Filmer; Filmer; Filmer;
Filmer; Filmer; Filmer; Filmer; Burgess; Burgess; Burgess; Morris;
Morris; Walker; Walker; Dalloway; Walker; Walker; Walker; Parkinson;
Barnet; Barnet; Barnet; Barnet; Barnet; Garrod; Hilbery; Mount;
Dakers; Durrant; Hilbery; Hilbery; Dalloway; Dalloway; Dalloway;
Dalloway; Hilbery; Hilbery

6. nltk book: chapter 3
nltk Text: .collocations() and .concordance(word)

7. nltk book: chapter 3

8. nltk book: chapter 3
>>> from nltk.corpus import brown >>> hobbies_learned = nltk.Text(brown.words(categories=['hobbies', 'learned'])) >>> hobbies_learned.findall(r"<\w*> <and> <other> <\w*s>") speed and other activities; water and other liquids; tomb and other landmarks; Statues and other monuments; pearls and other jewels; charts and other items; roads and other features; figures and other objects; military and other areas; demands and other factors; abstracts and other compilations; iron and other metals

9. nltk book: chapter 3 3.5 Useful Applications of Regular Expressions
Extracting Word Pieces
Relative frequency of sequences of two or more vowels
>>> import nltk
>>> import re
>>> from nltk.corpus import ptb
>>> t = ptb.words(categories='news')
>>> w = set(t)                                                                 
>>> len(w)
49817
>>> len(t)
>>> fd = nltk.FreqDist(vvs for word in w for vvs in re.findall(r'[aeiou]{2,}', word))
>>> fd.most_common(10)
[('io', 2090), ('ea', 1882), ('ou', 1421), ('ie', 1418), ('ia', 1083), ('ai', 970), ('ee', 874), ('oo', 783), ('au', 448), ('ei', 442)]
>>> fd.plot()
ptb: you don't have access to the full treebank (in this course)
treebank: use this instead

10. nltk book: chapter 3
iou
eau

11. nltk book: chapter 3
Readability: leave out word-internal vowels

12. nltk book: chapter 3 Word stemming
book example from Monty Python And The Holy Grail
/watch?v=eXmwK2-R2dY

13. nltk book: chapter 3

14. nltk book: chapter 3
Class IndexedText defined next slide:

15. nltk book: chapter 3
Advanced:
generates (index, word) tuples

16. nltk book: chapter 3
>>> from nltk import word_tokenize >>> raw = """DENNIS: Listen, strange women lying in ponds distributing swords ... is no basis for a system of government. Supreme executive power derives from ... a mandate from the masses, not from some farcical aquatic ceremony.""" >>> tokens = word_tokenize(raw) >>> wnl = nltk.WordNetLemmatizer() >>> [wnl.lemmatize(t) for t in tokens] ['DENNIS', ':', 'Listen', ',', 'strange', 'woman', 'lying', 'in', 'pond', 'distributing', 'sword', 'is', 'no', 'basis', 'for', 'a', 'system', 'of', 'government', '.', 'Supreme', 'executive', 'power', 'derives', 'from', 'a', 'mandate', 'from', 'the', 'mass', ',', 'not', 'from', 'some', 'farcical', 'aquatic', 'ceremony', '.']

17. nltk book: chapter 3 3.7 Regular Expressions for Tokenizing Text
word_tokenize() we take for granted. Simple tokenization:
>>> raw
"'When I'M a Duchess,' she said to herself, (not in a very hopeful tone\nthough), 'I won't have any pepper in my kitchen AT ALL. Soup does very\nwell without--Maybe it's always pepper that makes people hot-tempered,'..."
>>> import re
>>> re.split(r' ', raw)
["'When", "I'M", 'a', "Duchess,'", 'she', 'said', 'to', 'herself,', '(not', 'in', 'a', 'very', 'hopeful', 'tone\nthough),', "'I", "won't", 'have', 'any', 'pepper', 'in', 'my', 'kitchen', 'AT', 'ALL.', 'Soup', 'does', 'very\nwell', 'without--Maybe', "it's", 'always', 'pepper', 'that', 'makes', 'people', "hot-tempered,'..."]
>>> re.split(r'\s', raw) any whitespace
["'When", "I'M", 'a', "Duchess,'", 'she', 'said', 'to', 'herself,', '(not', 'in', 'a', 'very', 'hopeful', 'tone', 'though),', "'I", "won't", 'have', 'any', 'pepper', 'in', 'my', 'kitchen', 'AT', 'ALL.', 'Soup', 'does', 'very', 'well', 'without--Maybe', "it's", 'always', 'pepper', 'that', 'makes', 'people', "hot-tempered,'..."]

18. nltk book: chapter 3 Split on non-word characters:
>>> re.split(r'\W+', raw)
['', 'When', 'I', 'M', 'a', 'Duchess', 'she', 'said', 'to', 'herself', 'not', 'in', 'a', 'very', 'hopeful', 'tone', 'though', 'I', 'won', 't', 'have', 'any', 'pepper', 'in', 'my', 'kitchen', 'AT', 'ALL', 'Soup', 'does', 'very', 'well', 'without', 'Maybe', 'it', 's', 'always', 'pepper', 'that', 'makes', 'people', 'hot', 'tempered', '']
>>> re.findall(r'\w+|\S\w*', raw) Punctuation
["'When", 'I', "'M", 'a', 'Duchess', ',', "'", 'she', 'said', 'to', 'herself', ',', '(not', 'in', 'a', 'very', 'hopeful', 'tone', 'though', ')', ',', "'I", 'won', "'t", 'have', 'any', 'pepper', 'in', 'my', 'kitchen', 'AT', 'ALL', '.', 'Soup', 'does', 'very', 'well', 'without', '-', '-Maybe', 'it', "'s", 'always', 'pepper', 'that', 'makes', 'people', 'hot', '-tempered', ',', "'", '.', '.', '.']

19. word (possibly hyphenated)
nltk book: chapter 3
re.findall(r"\w+(?:[-']\w+)*|'|[-.(]+|\S\w*", raw)
["'", 'When', "I'M", 'a', 'Duchess', ',', "'", 'she', 'said', 'to', 'herself', ',', '(', 'not', 'in', 'a', 'very', 'hopeful', 'tone', 'though', ')', ',', "'", 'I', "won't", 'have', 'any', 'pepper', 'in', 'my', 'kitchen', 'AT', 'ALL', '.', 'Soup', 'does', 'very', 'well', 'without', '--', 'Maybe', "it's", 'always', 'pepper', 'that', 'makes', 'people', 'hot-tempered', ',', "'", '...']
(?: ….) means non-capturing group
word (possibly hyphenated)
apostrophe
-- or …
punctuation

20. nltk book: chapter 3 Remind ourselves what word_tokenize() does for ':
>>> word_tokenize(raw)
["'When", 'I', "'M", 'a', 'Duchess', ',', "'", 'she', 'said', 'to', 'herself', ',', '(', 'not', 'in', 'a', 'very', 'hopeful', 'tone', 'though', ')', ',', "'I", 'wo', "n't", 'have', 'any', 'pepper', 'in', 'my', 'kitchen', 'AT', 'ALL', '.', 'Soup', 'does', 'very', 'well', 'without', '--', 'Maybe', 'it', "'s", 'always', 'pepper', 'that', 'makes', 'people', 'hot-tempered', ',', "'", '...']

21. nltk book: chapter 3
>>> text = 'That U.S.A. poster-print costs $ '
>>> word_tokenize(text)
['That', 'U.S.A.', 'poster-print', 'costs', '$', '12.40', '...']