LING 388: Computers and Language Lecture 25

nltk book: chapter 3 Last time, we discussed the problem of word tokenization… >>> text = 'That U.S.A. poster-print costs $12.40...' >>> word_tokenize(text) ['That', 'U.S.A.', 'poster-print', 'costs', '$', '12.40', '...']

nltk book: chapter 3 3.8 Segmentation Sentence segmentation Brown corpus (pre-segmented): >>> len(nltk.corpus.brown.words()) / len(nltk.corpus.brown.sents()) 20.250994070456922 (average sentence length in terms of number of words) >>> raw = "'When I'M a Duchess,' she said to herself, (not in a very hopeful tone\nthough), 'I won't have any pepper in my kitchen AT ALL. Soup does very\nwell without--Maybe it's always pepper that makes people hot-tempered,'..." >>> nltk.sent_tokenize(raw) ["'When I'M a Duchess,' she said to herself, (not in a very hopeful tone\nthough), 'I won't have any pepper in my kitchen AT ALL.", "Soup does very\nwell without--Maybe it's always pepper that makes people hot-tempered,'..."] >>> nltk.sent_tokenize(raw)[0]                                                                   "'When I'M a Duchess,' she said to herself, (not in a very hopeful tone\nthough), 'I won't have any pepper in my kitchen AT ALL." >>> nltk.sent_tokenize(raw)[1]                                                                   "Soup does very\nwell without--Maybe it's always pepper that makes people hot-tempered,'..."

nltk book: chapter 3 Mrs. Dalloway revisited: >>> from urllib import request >>> url = "http://gutenberg.net.au/ebooks02/0200991.txt" >>> response = request.urlopen(url)  >>> raw = response.read().decode('latin-1')  >>> raw = raw[431:] >>> raw = raw[1217:] >>> raw = raw[:368976] >>> raw[:100] 'Title:      Mrs. Dalloway\r\nAuthor:     Virginia Woolf\r\n\r\n\r\n\r\n\r\nMrs. Dalloway said she would buy the ' >>> raw[-100:] 's me with extraordinary excitement?\r\n\r\nIt is Clarissa, he said.\r\n\r\nFor there she was.\r\n\r\n\r\n\r\nTHE END'

nltk book: chapter 3 >>> sents = nltk.sent_tokenize(raw) 0. 'Title:      Mrs. Dalloway\r\nAuthor:     Virginia Woolf\r\n\r\n\r\n \r\n\r\nMrs. Dalloway said she would buy the flowers herself.' 'For Lucy had her work cut out for her.' "The doors would be taken\r\noff their hinges; Rumpelmayer's men were coming." 'And then, thought\r\nClarissa Dalloway, what a morning--fresh as if issued to children\r\non a beach.' 'What a lark!' 'What a plunge!' 'For so it had always seemed to her,\r\nwhen, with a little squeak of the hinges, which she could hear now,\r\nshe had burst open the French windows and plunged at Bourton into\r\nthe open air.'

nltk book: chapter 3 Famous for her stream-of-consciousness style of writing: >>> sents[7] 'How fresh, how calm, stiller than this of course,\r\nthe air was in the early morning; like the flap of a wave; the kiss\r\nof a wave; chill and sharp and yet (for a girl of eighteen as she\r\nthen was) solemn, feeling as she did, standing there at the open\r\nwindow, that something awful was about to happen; looking at the\r\nflowers, at the trees with the smoke winding off them and the rooks\r\nrising, falling; standing and looking until Peter Walsh said,\r\n"Musing among the vegetables?"' >>> s7 = word_tokenize(sents[7]) >>> len(s7) 107 cf. Brown corpus average of 20 words/sentence

Python save/restore corpus json can be used (and is more standard across programming languages), but pickle is a Python library for this purpose: >>> import pickle >>> f = open('dalloway.pickle','wb') wb = write binary >>> pickle.dump(raw,f) >>> f.close() >>> f = open('dalloway.pickle','rb') rb = read binary >>> raw2 = pickle.load(f) >>> raw == raw2 True

nltk book: chapter 3 Python formatted output (printed as just one column, not three…) Python formatted output >>> import nltk >>> import pickle >>> f = open('dalloway.pickle','rb') >>> raw = pickle.load(f) >>> f.close() >>> fd = nltk.FreqDist(nltk.word_tokenize(raw)) >>> t50 = fd.most_common(50) >>> m = max(len(t[0]) for t in t50) >>> m 8 >>> for x,y in t50: ...     print('{:{width}} {}'.format(x,y,width=m)) ...  , 6098 . 3017 the 3015 and 1625 of 1525 ; 1473 to 1447 a 1328 was 1254 her 1227 she 1157 in 1107 had 928 he 908 it 712 that 622 with 565 --       545 his      490 ''       458 for      446 on       441 at       427 him      421 said     410 not      403 as       396 ``       388 She      372 ?        361 !        346 one      317 's       306 all      305 they     305 (        290 )        290 would    278 were     276 But      270 He       269 so       266 which    266 could    264 Clarissa 263 this     254 thought  252 be       245 up       232 like     232

nltk book: chapter 3

nltk book: chapter 3 At the terminal, line wrap is quite arbitrary… >>> from textwrap import fill >>> pieces = ['{} ({})'.format(t[0],t[1]) for t in t50] >>> s50 = ' '.join(pieces) >>> print(fill(s50)) , (6098) . (3017) the (3015) and (1625) of (1525) ; (1473) to (1447) a (1328) was (1254) her (1227) she (1157) in (1107) had (928) he (908) it (712) that (622) with (565) -- (545) his (490) '' (458) for (446) on (441) at (427) him (421) said (410) not (403) as (396) `` (388) She (372) ? (361) ! (346) one (317) 's (306) all (305) they (305) ( (290) ) (290) would (278) were (276) But (270) He (269) so (266) which (266) could (264) Clarissa (263) this (254) thought (252) be (245) up (232) like (232) fill() Text is preferably wrapped on whitespaces and right after the hyphens in hyphenated words; only then will long words be broken if necessary

nltk book: chapter 3 More fancy formatting: >>> pieces = ['{}_({})'.format(t[0],t[1]) for t in t50] >>> s50 = ' '.join(pieces) >>> print(fill(s50).replace('_',' ')) , (6098) . (3017) the (3015) and (1625) of (1525) ; (1473) to (1447) a (1328) was (1254) her (1227) she (1157) in (1107) had (928) he (908) it (712) that (622) with (565) -- (545) his (490) '' (458) for (446) on (441) at (427) him (421) said (410) not (403) as (396) `` (388) She (372) ? (361) ! (346) one (317) 's (306) all (305) they (305) ( (290) ) (290) would (278) were (276) But (270) He (269) so (266) which (266) could (264) Clarissa (263) this (254) thought (252) be (245) up (232) like (232)

nltk book: chapter 4 4 Writing Structured Programs teaches Python with many cool examples relevant to text processing Example: find the longest words in Milton's Paradise Lost >>> text = nltk.corpus.gutenberg.words('milton-paradise.txt') >>> maxlen = max(len(word) for word in text) >>> [word for word in text if len(word) == maxlen] ['unextinguishable', 'transubstantiate', 'inextinguishable', 'incomprehensible']

nltk book: chapter 4 Advanced topic: Generators (functions with yield instead of return)

nltk book: chapter 4 Brown corpus

nltk book: chapter 4

nltk book: chapter 4 WordNet

nltk book: chapter 4