Statistical Natural Language Processing

What is NLP?  Natural Language Processing (NLP), or Computational Linguistics, is concerned with theoretical and practical issues in the design and implementation of computer systems for processing human languages  It is an interdisciplinary field which draws on other areas of study such as computer science, artificial intelligence, linguistics and logic

Applications of NLP  natural language interfaces to databases  programs for classifying and retrieving documents by content  explanation generation for expert systems  machine translation  advanced word-processing tools

What makes NLP a computational challenge?  Ambiguous nature of Natural Language.  There are varied applications for language technology  Knowledge representation is a difficult task.  There are different levels of information encoded in our language

What is statistical NLP?  Statistical NLP aims to perform statistical inference for the field of NLP  Statistical inference consists of taking some data generated in accordance with some unknown probability distribution and making inferences.

Motivations for Statistical NLP  Cognitive modeling of the human language processing has not reached a stage where we can have a complete mapping between the language signal and the information contents.  Complete mapping is not always required.  Statistical approach provides the flexibility required for making the modeling of a language more accurate.

Idea behind Statistical NLP  View language processing as a noisy channel information transmission.  The approach requires a model that characterizes the transmission by giving for every message the probability of the observed output

Statistical Modeling and Classification  Primitive acoustic features  Quantization  Maximum likelihood and related rules  Class conditional density function  Hidden Markov Model Methodology

Details…. Primitive acoustic features are used to estimate the speech spectrum on the basis of its statistical properties. By means of quantization a typical speech signal can be represented as a sequence of symbols and can be mapped using statistical decision rules into a multidimensional acoustic feature space, thus classifying the signal.

Maximum Likelihood Although there is no direct method for computing the probability of a phonetic unit given its acoustic features,we can use Bayes rule to estimate the probability of a phonetic class given its features from the likelihood of the features given the class. This method leads to the maximum likelihood classifier which assigns an unknown vector to that class whose probability density function conditioned on the class has the maximum value. Another variant of the maximum likelihood methodology is clustering.

Hidden Markov Models A Hidden Markov Model, is a set of states (lexical categories in our case) with directed edges labeled with transition probabilities that indicate the probability of moving to the state at the end of the directed edge, given that one is now in the state at the start of the edge. The states are also labeled with a function which indicates the probabilities of outputting different symbols if in that state (while in a state, one outputs a single symbol before moving to the next state). In our case, the symbol output from a state/lexical category is a word belonging to that lexical category.

Hidden Markov Models (cont.)

Conditional Class Density Function All statistical methods of speech recognition depend on the class conditional density function. These, in turn, depend on the existence of a sufficiently large, correctly labeled training set and well understood statistical estimation techniques

How does statistics help  Disambiguation may be achieved by using stochastic context free grammars  It helps in providing degrees of grammaticality  Naturalness  Structural preference  Error Tolerance

Example using stochastic CFG for example consider the sentence “ John Walks “ The grammar is as follows : 1 S -> NP V S -> NP NP -> N NP -> N N N -> John N -> Walks V -> Walks 1.0 The numbers on the right represent the weights for each rule.The weight of the analysis is the product of the weights of the rules used in the derivation. Predicting the right sentence that is perceived is based on these weights.

Degrees of grammaticality  Traditional approaches to NLP do not accommodate gradations of grammaticality. A sentence is either correct or not.  In some cases acceptability may vary with the structure and context of the sentence.

Structural Preference Consider the sentence “ The emergency crews hate most is domestic violence.” The correct interpretation is: “The emergency [that the crews hate most] is domestic violence.” These preferences can be seen more as structural preferences rather than parsing preferences. Statistical approaches can easily handle such structural preferences.

Error Tolerance  A remarkable property of human language comprehension is error tolerance.  Many sentences that the traditional approach classifies as ungrammatical can actually be interpreted by statistical NLP techniques.

Conclusions  Free and commercial software is now available that provides a lot of NLP features. (e.g. Microsoft XP has a speech recognition software by which users can control menus and execute commands)  A lot of research is going into developing new applications and investigating new techniques and approaches that will make Statistical NLP more feasible in the near future.