1. Albert Gatt Corpora and Statistical Methods Lecture 11

2. Probabilistic Context-Free Grammars and beyond Part 1

3. Context-free grammars: reminder Many NLP parsing applications rely on the CFG formalism Definition: CFG is a 4-tuple: (N, Σ,P,S): N = a set of non-terminal symbols (e.g. NP, VP) Σ = a set of terminals (e.g. words) N and Σ are disjoint P = a set of productions of the form A  β A Є N β Є (N U Σ )* (any string of terminals and non-terminals) S = a designated start symbol (usually, “sentence”)

4. CFG Example S  NP VP S  Aux NP VP NP  Det Nom NP  Proper-Noun Det  that | the | a …

5. Probabilistic CFGs A CFG where each production has an associated probability PCFG is a 5-tuple: (N, Σ,P,S, D): D: P -> [0,1] a function assigning each rule in P a probability usually, probabilities are obtained from a corpus most widely used corpus is the Penn Treebank

6. The Penn Treebank English sentences annotated with syntax trees built at the University of Pennsylvania 40,000 sentences, about a million words text from the Wall Street Journal Other treebanks exist for other languages (e.g. NEGRA for German)

7. Example tree

8. Building a tree: rules S  NP VP NP  NNP NNP NNP  Mr NNP  Vinken … S NP NNP VinkenMr VP NP VBZ PPNP NN is chairman IN NN NNPof Elsevier

9. Characteristics of PCFGs In a PCFG, the probability P(A  β ) expresses the likelihood that the non- terminal A will expand as β. e.g. the likelihood that S  NP VP (as opposed to S  VP, or S  NP VP PP, or… ) can be interpreted as a conditional probability: probability of the expansion, given the LHS non-terminal P(A  β ) = P(A  β |A) Therefore, for any non-terminal A, probabilities of every rule of the form A  β must sum to 1 If this is the case, we say the PCFG is consistent

10. Uses of probabilities in parsing Disambiguation: given n legal parses of a string, which is the most likely? e.g. PP-attachment ambiguity can be resolved this way Speed: parsing is a search problem search through space of possible applicable derivations search space can be pruned by focusing on the most likely sub-parses of a parse Parser can be used as a model to determine the probability of a sentence, given a parse typical use in speech recognition, where input utterance can be “heard” as several possible sentences

11. Using PCFG probabilities PCFG assigns a probability to every parse-tree t of a string W e.g. every possible parse (derivation) of a sentence recognised by the grammar Notation: G = a PCFG s = a sentence t = a particular tree under our grammar t consists of several nodes n each node is generated by applying some rule r

12. Probability of a tree vs. a sentence simply the multiplication of the probability of every rule (node) that gives rise to t (i.e. the derivation of t) this is both the joint probability of t and s, and the probability of t alone why?

13. P(t,s) = P(t) But P(s|t) must be 1, since the tree t is a parse of all the words of s

14. Picking the best parse in a PCFG A sentence will usually have several parses we usually want them ranked, or only want the n-best parses we need to focus on P(t|s,G) probability of a parse, given our sentence and our grammar definition of the best parse for s:

15. Picking the best parse in a PCFG Problem: t can have multiple derivations e.g. expand left-corner nodes first, expand right-corner nodes first etc so P(t|s,G) should be estimated by summing over all possible derivations Fortunately, derivation order makes no difference to the final probabilities. can assume a “canonical derivation” d of t P(t) = def P(d)

16. Probability of a sentence Simply the sum of probabilities of all parses of that sentence since s is only a sentence if it’s recognised by G, i.e. if there is some t for s under G all those trees which “yield” s

17. Flaws I: Structural independence Probability of a rule r expanding node n depends only on n. Independent of other non-terminals Example: P(NP  Pro) is independent of where the NP is in the sentence but we know that NP  Pro is much more likely in subject position Francis et al (1999) using the Switchboard corpus: 91% of subjects are pronouns; only 34% of objects are pronouns

18. Flaws II: lexical independence vanilla PCFGs ignore lexical material e.g. P(VP  V NP PP) independent of the head of NP or PP or lexical head V Examples: prepositional phrase attachment preferences depend on lexical items; cf: dump [sacks into a bin] dump [sacks] [into a bin] (preferred parse) coordination ambiguity: [dogs in houses] and [cats] [dogs] [in houses and cats]

19. Weakening the independence assumptions in PCFGs

20. Lexicalised PCFGs Attempt to weaken the lexical independence assumption. Most common technique: mark each phrasal head (N,V, etc) with the lexical material this is based on the idea that the most crucial lexical dependencies are between head and dependent E.g.: Charniak 1997, Collins 1999

21. Lexicalised PCFGs: Matt walks Makes probabilities partly dependent on lexical content. P(VP  VBD|VP) becomes: P(VP  VBD|VP, h(VP)=walk) NB: normally, we can’t assume that all heads of a phrase of category C are equally probable. S(walks) NP(Matt) NNP(Matt) Matt VP(walk) VBD(walk) walks

22. Practical problems for lexicalised PCFGs data sparseness: we don’t necessarily see all heads of all phrasal categories often enough in the training data flawed assumptions: lexical dependencies occur elsewhere, not just between head and complement I got the easier problem of the two to solve of the two and to solve become more likely because of the prehead modifier easier

23. Structural context The simple way: calculate p(t|s,G) based on rules in the canonical derivation d of t assumes that p(t) is independent of the derivation could condition on more structural context but then we could lose the notion of a canonical derivation, i.e. P(t) could really depend on the derivation!

24. Structural context: probability of a derivation history How to calculate P(t) based on a derivation d? Observation: (probability that a sequence of m rewrite rules in a derivation yields s) can use the chain rule for multiplication

25. Approach 2: parent annotation Annotate each node with its parent in the parse tree. E.g. if NP has parent S, then rename NP to NP^S Can partly account for dependencies such as subject-of (NP^S is a subject, NP^VP is an object) S(walks) NP^S NNP^NP Matt VP^S VBD^VP walks

26. The main point Many different parsing approaches differ on what they condition their probabilities on

27. Other grammar formalisms

28. Phrase structure vs. Dependency grammar PCFGs are in the tradition of phrase-structure grammars Dependency grammar describes syntax in terms of dependencies between words no non-terminals or phrasal nodes only lexical nodes with links between them links are labelled, labels from a finite list

29. Dependency Grammar GAVE main I him address subj: dat: obj: MY attr:

30. Dependency grammar Often used now in probabilistic parsing Advantages: directly encode lexical dependencies therefore, disambiguation decisions take lexical material into account directly dependencies are a way of decomposing PSRs and their probability estimates estimating probability of dependencies between 2 words is less likely to lead to data sparseness problems

31. Summary We’ve taken a tour of PCFGs crucial notion: what the probability of a rule is conditioned on flaws in PCFGs: independence assumptions several proposals to go beyond these flaws dependency grammars are an alternative formalism