Statistical Natural Language Processing Advanced AI - Part II Luc De Raedt University of Freiburg WS 2005/2006 Many slides taken from Helmut Schmid.

Slides:



Advertisements
Similar presentations
Grammar is to Meaning as the Law if to Good Behaviour Adam Kilgarriff Lexical Computing Ltd Lexicography MasterClass Ltd Universities of Leeds and Sussex.
Advertisements

1 CS 388: Natural Language Processing: N-Gram Language Models Raymond J. Mooney University of Texas at Austin.
Stochastic Context Free Grammars for RNA Modeling CS 838 Mark Craven May 2001.
10. Lexicalized and Probabilistic Parsing -Speech and Language Processing- 발표자 : 정영임 발표일 :
1 Statistical NLP: Lecture 12 Probabilistic Context Free Grammars.
March 1, 2009 Dr. Muhammed Al-Mulhem 1 ICS 482 Natural Language Processing Probabilistic Context Free Grammars (Chapter 14) Muhammed Al-Mulhem March 1,
January 12, Statistical NLP: Lecture 2 Introduction to Statistical NLP.
Applied Computer Science II Chapter 1 : Regular Languages Prof. Dr. Luc De Raedt Institut für Informatik Albert-Ludwigs Universität Freiburg Germany.
Natural Language and Speech Processing Creation of computational models of the understanding and the generation of natural language. Different fields coming.
Advanced AI - Part II Luc De Raedt University of Freiburg WS 2004/2005 Many slides taken from Helmut Schmid.
Part II. Statistical NLP Advanced Artificial Intelligence Probabilistic Context Free Grammars Wolfram Burgard, Luc De Raedt, Bernhard Nebel, Lars Schmidt-Thieme.
Part II. Statistical NLP Advanced Artificial Intelligence Introduction and Grammar Models Wolfram Burgard, Luc De Raedt, Bernhard Nebel, Kristian Kersting.
Applied Computer Science II Chapter : Grammars Prof. Dr. Luc De Raedt Institut für Informatik Albert-Ludwigs Universität Freiburg Germany.
Part II. Statistical NLP Advanced Artificial Intelligence Part of Speech Tagging Wolfram Burgard, Luc De Raedt, Bernhard Nebel, Lars Schmidt-Thieme Most.
Applied Computer Science II Chapter 2 : Context-free languages Prof. Dr. Luc De Raedt Institut für Informatik Albert-Ludwigs Universität Freiburg Germany.
CS 4705 Lecture 13 Corpus Linguistics I. From Knowledge-Based to Corpus-Based Linguistics A Paradigm Shift begins in the 1980s –Seeds planted in the 1950s.
CSE 574 – Artificial Intelligence II Statistical Relational Learning Instructor: Pedro Domingos.
Big Ideas in Cmput366. Search Blind Search Iterative deepening Heuristic Search A* Local and Stochastic Search Randomized algorithm Constraint satisfaction.
1/13 Parsing III Probabilistic Parsing and Conclusions.
Grammar induction by Bayesian model averaging Guy Lebanon LARG meeting May 2001 Based on Andreas Stolcke’s thesis UC Berkeley 1994.
تمرين شماره 1 درس NLP سيلابس درس NLP در دانشگاه هاي ديگر ___________________________ راحله مکي استاد درس: دکتر عبدالله زاده پاييز 85.
CPSC 322, Lecture 31Slide 1 Probability and Time: Markov Models Computer Science cpsc322, Lecture 31 (Textbook Chpt 6.5) March, 25, 2009.
Machine Learning in Natural Language Processing Noriko Tomuro November 16, 2006.
UNIVERSITY OF SOUTH CAROLINA Department of Computer Science and Engineering Using Definite Knowledge: NLP and nl_interface.pl Notes for Ch.3 of Poole et.
Statistical Natural Language Processing. What is NLP?  Natural Language Processing (NLP), or Computational Linguistics, is concerned with theoretical.
Lecture 1, 7/21/2005Natural Language Processing1 CS60057 Speech &Natural Language Processing Autumn 2005 Lecture 1 21 July 2005.
9/8/20151 Natural Language Processing Lecture Notes 1.
Part II. Statistical NLP Advanced Artificial Intelligence Applications of HMMs and PCFGs in NLP Wolfram Burgard, Luc De Raedt, Bernhard Nebel, Lars Schmidt-Thieme.
6. N-GRAMs 부산대학교 인공지능연구실 최성자. 2 Word prediction “I’d like to make a collect …” Call, telephone, or person-to-person -Spelling error detection -Augmentative.
Information Retrieval and Web Search Text properties (Note: some of the slides in this set have been adapted from the course taught by Prof. James Allan.
Natural Language Processing (NLP) I. Introduction II. Issues in NLP III. Statistical NLP: Corpus-based Approach.
10/12/2015CPSC503 Winter CPSC 503 Computational Linguistics Lecture 10 Giuseppe Carenini.
1 Chapter 3 Describing Syntax and Semantics. 3.1 Introduction Providing a concise yet understandable description of a programming language is difficult.
1 Statistical NLP: Lecture 9 Word Sense Disambiguation.
UCREL: from LOB to REVERE Paul Rayson. November 1999CSEG awayday Paul Rayson2 A brief history of UCREL In ten minutes, I will present a brief history.
Introduction to CL & NLP CMSC April 1, 2003.
CSCI 6900/4900 Special Topics in Computer Science Automata and Formal Grammars for Bioinformatics Bioinformatics problems sequence comparison pattern/structure.
C H A P T E R TWO Syntax and Semantic.
ISBN Chapter 3 Describing Syntax and Semantics.
인공지능 연구실 황명진 FSNLP Introduction. 2 The beginning Linguistic science 의 4 부분 –Cognitive side of how human acquire, produce, and understand.
11 Chapter 14 Part 1 Statistical Parsing Based on slides by Ray Mooney.
Natural Language Processing Spring 2007 V. “Juggy” Jagannathan.
1 CSI 5180: Topics in AI: Natural Language Processing, A Statistical Approach Instructor: Nathalie Japkowicz Objectives of.
October 2005CSA3180 NLP1 CSA3180 Natural Language Processing Introduction and Course Overview.
1 CS 385 Fall 2006 Chapter 1 AI: Early History and Applications.
AdvancedBioinformatics Biostatistics & Medical Informatics 776 Computer Sciences 776 Spring 2002 Mark Craven Dept. of Biostatistics & Medical Informatics.
Tokenization & POS-Tagging
Chapter 23: Probabilistic Language Models April 13, 2004.
Language and Statistics
Statistical Decision-Tree Models for Parsing NLP lab, POSTECH 김 지 협.
Introduction Chapter 1 Foundations of statistical natural language processing.
For Friday Finish chapter 23 Homework –Chapter 23, exercise 15.
CPSC 422, Lecture 27Slide 1 Intelligent Systems (AI-2) Computer Science cpsc422, Lecture 27 Nov, 16, 2015.
SIMS 296a-4 Text Data Mining Marti Hearst UC Berkeley SIMS.
CS : Speech, NLP and the Web/Topics in AI Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture-15: Probabilistic parsing; PCFG (contd.)
N-Gram Model Formulas Word sequences Chain rule of probability Bigram approximation N-gram approximation.
CS460/626 : Natural Language Processing/Speech, NLP and the Web (Lecture 25– Probabilistic Parsing) Pushpak Bhattacharyya CSE Dept., IIT Bombay 14 th March,
Chapter 3 – Describing Syntax CSCE 343. Syntax vs. Semantics Syntax: The form or structure of the expressions, statements, and program units. Semantics:
Describing Syntax and Semantics Chapter 3: Describing Syntax and Semantics Lectures # 6.
Text Based Information Retrieval
What does it mean? Notes from Robert Sebesta Programming Languages
Machine Learning in Natural Language Processing
Language and Statistics
Stochastic Context Free Grammars for RNA Structure Modeling
CS4705 Natural Language Processing
Natural Language Processing
EA C461 – Artificial Intelligence Introduction
Lecture 13 Corpus Linguistics I CS 4705.
PRESENTATION: GROUP # 5 Roll No: 14,17,25,36,37 TOPIC: STATISTICAL PARSING AND HIDDEN MARKOV MODEL.
Presentation transcript:

Statistical Natural Language Processing Advanced AI - Part II Luc De Raedt University of Freiburg WS 2005/2006 Many slides taken from Helmut Schmid

Topic Statistical Natural Language Processing Applies  Machine Learning / Statistics to Learning : the ability to improve one’s behaviour at a specific task over time - involves the analysis of data (statistics)  Natural Language Processing Following parts of the book  Statistical NLP (Manning and Schuetze), MIT Press, 1999.

Rationalism versus Empiricism Rationalist  Noam Chomsky - innate language structures  AI : hand coding NLP  Dominant view  Cf. e.g. Steven Pinker’s The language instinct. (popular science book) Empiricist  Ability to learn is innate  AI : language is learned from corpora  Dominant and becoming increasingly important

Rationalism versus Empiricism Noam Chomsky:  But it must be recognized that the notion of “probability of a sentence” is an entirely useless one, under any known interpretation of this term Fred Jelinek (IBM 1988)  Every time a linguist leaves the room the recognition rate goes up.  (Alternative: Every time I fire a linguist the recognizer improves)

This course Empiricist approach  Focus will be on probabilistic models for learning of natural language No time to treat natural language in depth !  (though this would be quite useful and interesting)  Deserves a full course by itself Covered in more depth in Logic, Language and Learning (SS 05, prob. SS 06)

Ambiguity

Statistical Disambiguation Define a probability model for the data Compute the probability of each alternative Choose the most likely alternative NLP and Statistics

Statistical Methods deal with uncertainty. They predict the future behaviour of a system based on the behaviour observed in the past.  Statistical Methods require training data. The data in Statistical NLP are the Corpora NLP and Statistics

 Corpus: text collection for linguistic purposes  Tokens How many words are contained in Tom Sawyer?   Types How many different words are contained in T.S.?   Hapax Legomena words appearing only once Corpora

 The most frequent words are function words wordfreqwordfreq the3332in906 and2972that877 a1775he877 to1725I783 of1440his772 was1161you686 it1027Tom679 Word Counts

f n f > How many words appear f times? Word Counts About half of the words occurs just once About half of the text consists of the 100 most common words ….

Word Counts (Brown corpus)

wordfr f*rwordfr f*r the turned and you‘ll a name he comes but group be lead there friends one begin about family more brushed never sins Oh Could two Applausive Zipf‘s Law: f~1/r (f*r = const) Zipf‘s Law Minimize effort

Some probabilistic models N-grams  Predicting the next word  Artificial intelligence and machine ….  Statistical natural language …. Probabilistic  Regular (Markov Models)  Hidden Markov Models  Conditional Random Fields  Context-free grammars  (Stochastic) Definite Clause Grammars

Illustration Wall Street Journal Corpus words Correct parse tree for sentences known  Constructed by hand  Can be used to derive stochastic context free grammars  SCFG assign probability to parse trees Compute the most probable parse tree

Conclusions Overview of some probabilistic and machine learning methods for NLP Also very relevant to bioinformatics !  Analogy between parsing A sentence A biological string (DNA, protein, mRNA, …)