Download presentation
Presentation is loading. Please wait.
Published byGeorgiana McKinney Modified over 9 years ago
1
October 2005CSA3180: Text Processing I1 CSA3180: Natural Language Processing Text Processing 1 Language Encoding Issues Common Corpora Handling Large Document Collections Applications: Anatomy of a Search Engine NLTK
2
October 2005CSA3180: Text Processing I2 Language Encoding Issues Different encoding methods Different languages Unicode Standard Further information: –Unicode Consortium –Jukka Korpela Tutorial http://www.cs.tut.fi/~jkorpela/chars.html
3
October 2005CSA3180: Text Processing I3 Language Encoding Issues Character Repertoire – set of distinct characters Character Code – mapping between characters and positive integers Character Encoding – algorithm for presenting characters using particular code
4
October 2005CSA3180: Text Processing I4 Language Encoding Issues Encoding using octets Common Encodings: –ASCII –ISO Latin I (ISO 8859-1) –ISO Latin II + III Extensions (for Maltese) –Unicode & UTF-8 –ANSI –Cyrillic and Chinese Encodings
5
October 2005CSA3180: Text Processing I5 Language Encoding Issues Text encoding on the Web MIME Standard –Content-Type: text/html; charset=iso-8859-1 –Used in Email and Web Servers –Problems in implementation: few encodings properly supported –UTF-8 recommended
6
October 2005CSA3180: Text Processing I6 Common Corpora WordNet TREC/ACE/TIDES Corpora Linguistic Data Consortium (LDC) –GigaWord (News) –Tree Banks –MUC (Message Understanding Conference) –TIPSTER (Information Retrieval)
7
October 2005CSA3180: Text Processing I7 Handling Large Document Collections Special issues involved in processing Hierarchical directory structures File indexes Batch processing – start, resume, pause, end Job scheduling
8
October 2005CSA3180: Text Processing I8 Applications Anatomy of a Search Engine (Larry Page and Sergey Brin) Describes the internals of Google NLP in everyday life!
9
October 2005CSA3180: Text Processing I9 Next Sessions… Natural Language Toolkit (NLTK) http://nltk.sourceforge.net/ Please download and install!
Similar presentations
© 2024 SlidePlayer.com. Inc.
All rights reserved.