1. hfaili@ece.ut.ac.ir Introduction to Natural Language Processing Heshaam Faili hfaili@ece.ut.ac.ir

2. Session Agenda  Artificial Intelligence  Natural Language Processing  History of NLP  Statistical NLP  Applications of NLP

3. hfaili@ece.ut.ac.ir AI Concepts and Definitions Encompasses Many Definitions AI Involves Studying HumanThought Processes Representing Thought Processes on Machines

4. hfaili@ece.ut.ac.ir Artificial Intelligence Behavior by a machine that, if performed by a human being, would be considered intelligent “…study of how to make computers do things at which, at the moment, people are better” (Rich and Knight [1991]) Theory of how the human mind works (Mark Fox)

5. hfaili@ece.ut.ac.ir AI Objectives Make machines smarter (primary goal) Understand what intelligence is Make machines more useful (practical purpose) (Winston and Prendergast [1984])

6. hfaili@ece.ut.ac.ir Turing Test for Intelligence A computer can be considered to be smart only when a human interviewer, “conversing” with both an unseen human being and an unseen computer, can not determine which is which

7. hfaili@ece.ut.ac.ir Major AI Areas  Expert Systems  Natural Language Processing  Speech Understanding  Robotics and Sensory Systems  Computer Vision and Scene Recognition  Intelligent Computer-Aided Instruction  Neural Computing  Fuzzy Logic  Genetic Algorithms  Intelligent Software Agents

8. hfaili@ece.ut.ac.ir What is NLP ? Natural Language is one of fundamental aspects of human behaviors. One of the final aim of human- computer communication. Provide easy interaction with computer Make computer to understand texts.

9. hfaili@ece.ut.ac.ir Major Disciplines Studying Language DisciplineTypical Problem Linguists How do words from phrases and sentences? Psycholinguists How do people identify the structure of sentences? Philosophers What is meaning and how do words and sentences acquires it? Natural Language Processing How is the structure of sentences identified?

10. hfaili@ece.ut.ac.ir Interaction Level The level that computer and human interact. NL used for make Interaction level near to human. HumanComputer Command-line NL UI Graphical UI Interaction level

11. hfaili@ece.ut.ac.ir Other Titles The most common titles, apart from Natural Language Processing include: Automatic Language Processing Computational Linguistics Natural Language Understanding

12. hfaili@ece.ut.ac.ir Computational Linguistics  This is the application of computers to the scientific study of human language.  This definition suggests that there are connections with Cognitive Science, that is to say, the study of how humans produce and understand language.

13. hfaili@ece.ut.ac.ir Computational Linguistics  Historically, Computational Linguistics has been associated with work in Generative Linguistics and formerly included the study of formal languages (eg finite state automata) and programming languages.

14. hfaili@ece.ut.ac.ir Natural Language Understanding  Distinguish a particular approach to Natural Language Processing.  The people using this title tend to lay much emphasis on the meaning of the language being processed, in particular getting the computer to respond to the input in an apparently intelligent fashion.

15. hfaili@ece.ut.ac.ir Natural Language Understanding  At one time, those who belonged to the Natural Language Understanding camp avoided the use of any syntactic processing, but textbooks that bear this title now include significant sections on syntactic processing, which suggests that the edge of the title has been rather blunted. (For instance, see Allen (1987; part 1).Allen (1987

16. hfaili@ece.ut.ac.ir Motivation for NLP Understand language analysis & generation Communication Language is a window to the mind Data is in linguistic form Data can be in Structured (table form), Semi structured (XML form), Unstructured (sentence form).

17. hfaili@ece.ut.ac.ir Language Processing Level 1 – Speech sound (Phonetics & Phonology) Level 2 – Words & their forms (Morphology, Lexicon) Level 3 – Structure of sentences (Syntax, Parsing) Level 4 – Meaning of sentences (Semantics) Level 5 – Meaning in context & for a purpose (Pragmatics) Level 6 – Connected sentence processing in a larger body of text (Discourse)

18. hfaili@ece.ut.ac.ir Phonetics Concerns processing or identifying Languages Accents Pauses Word boundaries Amplitude, Tone Also includes background noise elimination E.g. “I got up late” and “I got a plate” sound similar

19. hfaili@ece.ut.ac.ir Lexicon Deals with vocabulary of words Uses Dictionary, Wordnet etc. Various levels of richness in dictionary, e.g. tense, senses, usage, etc. Resources – Princeton, Euro-wordnet, …

20. hfaili@ece.ut.ac.ir Syntax Involves parsing and understanding structure of grammar Challenges Ungrammatical sentences Word order – fixed, free Word attachment and scope e.g. Old men and women were rescued. Only old men or old women too Prepositional phrase attachment e.g. I saw the boy with a telescope With associated with boy or telescope?

21. hfaili@ece.ut.ac.ir Semantics Concerned with “meaning” Creates a structure for a sentence Main verb associated with agent, object, instrument, etc. E.g. I ate rice with spoon. instrument eat spoon I rice agent obj –Challenges Representation Domain (straddles into pragmatics) To construct meaning from individual meanings

22. hfaili@ece.ut.ac.ir Pragmatics Use of the sentence in a situation Understanding user's intention E.g. Is that water? response different on dining table and in chemistry lab Applications: Search engine tuned to user preferences

23. hfaili@ece.ut.ac.ir Discourse Processing of connected text Co-reference – Two expressions in the utterance, both refer to the same thing. Examples Pronoun to noun binding – John is sleeping. He is lazy (He refers to John) In an article – George Bush, Mr. Bush, The President of United States, The President General to specific – Ferrari launched a new model. This car is much better than the previous one. Car refers to new model launched

24. hfaili@ece.ut.ac.ir NLP History (1)  The first recognizable NLP application was a dictionary look-up system developed at Birkbeck College, London in 1948.

25. hfaili@ece.ut.ac.ir NLP History (2)  NLP from 1966-1980  Augmented Transition Networks  Case Grammar  Semantic representations  Conceptual Dependency  Semantic network  Procedural semantics

26. hfaili@ece.ut.ac.ir NLP History (3)  The key systems were:  LUNAR: A database interface system that used ATNs and Woods' Procedural Semantics.  LIFER/LADDER: One of the most impressive of NLP systems. It was designed as a natural language interface to a database of information about US Navy ships.  NLP from 1980 - 1990 - Grammar Formalisms  NLP from 1990- 2000 - Multilinguality and Multimodality  NLP from 2000-now - Statistical Approaches and Practical Uses

27. hfaili@ece.ut.ac.ir Why NLP is Hard?

28. hfaili@ece.ut.ac.ir Why NLP is Hard?

29. hfaili@ece.ut.ac.ir Why NLP is Hard?

30. hfaili@ece.ut.ac.ir Why NLP is Hard?

31. hfaili@ece.ut.ac.ir Why NLP is Hard?

32. hfaili@ece.ut.ac.ir Basics of statistical NLP Consider NLP problems as sequence labeling tasks Amenable to machine learning (training and generalization) In classical NLP – rules are obtained from linguists In statistical NLP – probabilities are learnt from data

33. hfaili@ece.ut.ac.ir Noisy Channel Metaphor SpeechText Signal - I want food. - It is cold today. Noisy

34. hfaili@ece.ut.ac.ir Data-Driven Approach The issues in this approach are - Corpora collection (coherent piece of text) Corpora cleaning – spelling, grammar, strange characters’ removal Annotation Named entity recognition POS detection Parsing Meaning Again: The biggest challenge is Ambiguity.

35. hfaili@ece.ut.ac.ir Sequence Labeling Tasks In the order of complexity - Dealing words – POS tagging, Named Entity Recognition (NER), Sense disambiguation Phrases – Chunking Sentences – Bracketing Paragraphs – Co-referencing

36. hfaili@ece.ut.ac.ir Examples of Levels Example Sentence – The dog Bill went near cat Jack. It bit it POS Tagging – The dog Bill went near cat Jack. It bit it DT NN NNP VBD PP NN NNP PN VBD PN NER – Bill Jack Sense – Using Wordnet {dog, animal} – synset-id synset-id assigned to each sense

37. hfaili@ece.ut.ac.ir Chunking (Beginning, Intermediate, End) (The dog Bill) went near (the cat Jack) BIE BIE BIE B IE It bit it BIE BIE BIE

38. hfaili@ece.ut.ac.ir Parsing S NPVP DTNP V PP N N PNP dogBill near went the the cat Jack

39. hfaili@ece.ut.ac.ir Higher Order Structures Bracketing – [S [NP] [VP [V [PP [P [NP]]]]]] [S [NP] [VP [V [NP]]]] Co-referencing The dog Bill went near the cat Jack. It bit it 12 3 4 56 78 9 10 11 References – 2<-9, 7<-11, 2<-3, 7<-8

40. hfaili@ece.ut.ac.ir Sequence labeling task is a classification task POS NER Sense Chunking Bracketing word->POS cat{NN, VBD...} word->Name cat{person, place} word->sense-id{001... N} word->{B, I, E} sentence->{has_tree, no_tree} TaskClassification

41. hfaili@ece.ut.ac.ir Learning Algorithm Knowledge Based Rules Decision Trees Decision Lists Statistical Graphical Models – HMM Neural Networks Support Vector Machines (SVM)

42. hfaili@ece.ut.ac.ir Applications Machine Translation: different strategies Systran: www.Systransoft.comwww.Systransoft.com Google: Translate.google.com Question – Answering MIT Q&A system( START ): http://start.csail.mit.edu/ Summarization: Information Extraction Spell Checking Microsoft Spell Checker Call centre MT for SMS

43. hfaili@ece.ut.ac.ir NLP Laboratory The first aim is to establish a virtual center for NLP related researches Defining of practical applications specially on Persian POS TAGGER, Spell Checker, n-gram model, Machine translation, NER, Document Classification, Search Engine, Summarization, Defining several research projects Sharing different resources and experiences Make a foundation of NLP-Suite Like TINA : MIT NLP-SUITE Contact me for any request on NLP domain (hfaili@ece.ut.ac.ir)

44. hfaili@ece.ut.ac.ir ?