1. Parsing Based on presentations from Chris Manning’s course on Statistical Parsing (Stanford)

2. S NVP VNP DN John hit theball

3. Levels of analysis LevelElements used Morphology/LexicalWords POS (morpho-synactic), WSDWords Shallow syntax parsingPhrases Full syntax parsingSentence NER, MWEPhrases SRLParsed trees Full semantic parsingParsed trees

4. Buffalo…

5. Parsing is a difficult task! ^______^ so excited! #Khaleesi #miniKhaleesi #GoT

6. Ambiguities  POS tags (e.g., books : a verb or a noun?)  Compositional expression meanings (e.g., he spilled the beans about his past)  Syntactic attachments (V N PP) (e.g., I ate my spaghettis with a fork)  Global semantic ambiguities (e.g., bear left at zoo) Usually, ambiguities in one layer may be resolved in upper layers

7. Ambiguities Fed raises interest rates 0.5 % in effort to control inflation

8. Motivation  Parsing may help to resolve ambiguities  Parsing is a step toward understanding the sentence completely  Was shown to improve the results of several NLP applications:  MT (Chiang, 2005)  Question answering (Hovy et al., 2000)  …

9. Grammar  S  NP VPNN  interest  NP  (DT) NNNNS  rates  NP  NN NNSNNS  raises  NP  NNPVBP  interest  VP  V NPVBZ  rates  …  Minimal grammar on “Fed raises” sentence: 36 parses  Simple 10 rule grammar: 592 parses  Real-size broad-coverage grammar: millions of parses

10. Size of grammar less more Number of rules Limits unlikely parses But grammar is not robust Parses more sentences But sentences end up with ever more parses

11. Statistical parsing Statistical parsing can help selecting the rules that best fit the input sentence, allowing the grammar to contain more rules

12. Treebanks ( (S (NP-SBJ (DT The) (NN move)) (VP (VBD followed) (NP (NP (DT a) (NN round)) (PP (IN of) (NP (NP (JJ similar) (NNS increases)) (PP (IN by) (NP (JJ other) (NNS lenders))) (PP (IN against) (NP (NNP Arizona) (JJ real) (NN estate) (NNS loans)))))) (,,) … The Penn Treebank Project (PTB): Arabic, English, Chinese, Persian, French,…

13. Advantages of treebanks  Reusability of the labor  Broad coverage  Frequencies and distributional information  A way to evaluate systems

14. Types of parsing Constituency parsingDependency parsing

15. Constituency parsing  Constituents are defined based on linguistic rules (phrases)  Constituents are recursive (NP may contain NP as part of its sub-constituents)  Different linguists may define constituents differently…

16. Dependency parsing Dependency structure shows which words depend on (modify or are arguments of) which other words

17. Parsing  We want to run a grammar backwards to find possible structures for a sentence  Parsing can be viewed as a search problem  We can do this bottom-up or top-down  We search by building a search tree which his distinct from the parse tree

18. Phrase structure grammars = context-free grammars (CFG)  G = (T, N, S, R)  T is set of terminals  N is set of nonterminals  S is the start symbol (one of the nonterminals)  R is rules/productions of the form X  , where X is a nonterminal and  is a sequence of terminals and nonterminals (possibly an empty sequence)  A grammar G generates a language L

19. Probabilistic or stochastic context-free grammars (PCFGs)  G = (T, N, S, R, P)  T is set of terminals  N is set of nonterminals  S is the start symbol (one of the nonterminals)  R is rules/productions of the form X  , where X is a nonterminal and  is a sequence of terminals and nonterminals (possibly an empty sequence)  P(R) gives the probability of each rule  A grammar G generates a language L

20. Soundness and completeness  A parser is sound if every parse it returns is valid/correct  A parser terminates if it is guaranteed to not go off into an infinite loop  A parser is complete if for any given grammar and sentence, it is sound, produces every valid parse for that sentence, and terminates (For many purposes, we settle for sound but incomplete parsers: e.g., probabilistic parsers that return a k-best list.)

21. Top down parsing  Top-down parsing is goal directed  A top-down parser starts with a list of constituents to be built. The top-down parser rewrites the goals in the goal list by matching one against the LHS of the grammar rules, and expanding it with the RHS, attempting to match the sentence to be derived  If a goal can be rewritten in several ways, then there is a choice of which rule to apply (search problem)  Can use depth-first or breadth-first search, and goal ordering

22. Top down parsing

23. Disadvantages of top down  A top-down parser will do badly if there are many different rules for the same LHS. Consider if there are 600 rules for S, 599 of which start with NP, but one of which starts with V, and the sentence starts with V  Useless work: expands things that are possible top- down but not there  Repeated work

24. Repeated work

25. Bottom up chart parsing  Bottom-up parsing is data directed  The initial goal list of a bottom-up parser is the string to be parsed. If a sequence in the goal list matches the RHS of a rule, then this sequence may be replaced by the LHS of the rule  Parsing is finished when the goal list contains just the start category  If the RHS of several rules match the goal list, then there is a choice of which rule to apply (search problem)  The standard presentation is as shift-reduce parsing

26. Shift-reduce parsing cats scratch people with claws catsscratch people with clawsSHIFT Nscratch people with clawsREDUCE NPscratch people with clawsREDUCE NP scratchpeople with clawsSHIFT NP Vpeople with clawsREDUCE NP V peoplewith clawsSHIFT NP V Nwith clawsREDUCE NP V NPwith clawsREDUCE NP V NP withclawsSHIFT NP V NP PclawsREDUCE NP V NP P clawsSHIFT NP V NP P NREDUCE NP V NP P NPREDUCE NP V NP PPREDUCE NP VPREDUCE SREDUCE

27. Disadvantages of bottom up  Useless work: locally possible, but globally impossible.  Inefficient when there is great lexical ambiguity (grammar- driven control might help here)  Repeated work: anywhere there is common substructure

28. Parsing as search  Left recursive structures must be found, not predicted  Doing these things doesn't fix the repeated work problem:  Both TD and BU parsers can (and frequently do) do work exponential in the sentence length on NLP problems  Grammar transformations can fix both left-recursion and epsilon productions  Then you parse the same language but with different trees (and fix them post hoc)

29. Dynamic programming  Rather than doing parsing-as-search, we do parsing as dynamic programming  Examples: CYK (bottom up), Early (top down)  It solves the problem of doing repeated work

30. Notation  w 1n = w 1 … w n = the word sequence from 1 to n  w ab = the subsequence w a … w b  N j ab = the nonterminal N j dominating w a … w b  We’ll write P(N i  ζ j ) to mean P(N i  ζ j | N i )  We’ll want to calculate max t P(t  * w ab )

31. Tree and sentence probabilities  P(t) -- The probability of tree is the product of the probabilities of the rules used to generate it  P(w 1n ) -- The probability of the sentence is the sum of the probabilities of the trees which have that sentence as their yield P(w 1n ) = Σ j P(w 1n, t) where t is a parse of w 1n = Σ j P(t)

32. Phrase structure grammars = context-free grammars (CFG)  G = (T, N, S, R)  T is set of terminals  N is set of nonterminals  S is the start symbol (one of the nonterminals)  R is rules/productions of the form X  , where X is a nonterminal and  is a sequence of terminals and nonterminals (possibly an empty sequence)  A grammar G generates a language L

33. Chomsky Normal Form (CNF)  All rules are of the form X  Y Z or X  w  A transformation to this form doesn’t change the generative capacity of CFG  With some extra book-keeping in symbol names, you can even reconstruct the same trees with a de-transform  Unaries/empties are removed recursively  N-ary rules introduce new non-terminals (binarization):  VP  V NP PP becomes VP  V @VP-V and @VP-V  NP PP  In practice it’s a pain  Reconstructing n-aries is easy  Reconstructing unaries can be trickier  But it makes parsing easier/more efficient

34. A treebank tree ROOT S NP VP N cats V NP PP PNP claws withpeople scratch N N

35. After binarization P NP claws N @PP->_P with NP N cats people scratch N VP V NP PP @VP->_V @VP->_V_NP ROOT S @S->_NP

36. CYK ( Cocke-Younger-Kasami) algorithm  A bottom-up parser using dynamic programming  Assume the PCFG is in Chomsky normal form (CNF)  Maintain |N| nXn tables µ (|N| = number of non- terminals, n = number of input words [length of input sentence])  Fill out the table entries by induction

37. “Can 1 you 2 book 3 ELAL 4 flights 5 ?” w1,1w1,2w1,3w1,4w1,5 w2,2w2,3w2,4w2,5 w3,3w3,4w3,5 w4,4w4,5 w5,5 1 23 45 1 2 3 4 5

38. CYK Base case – Consider the input strings of length one (i.e., each individual word w i ) P(A  w i ) – Since the grammar is in CNF: A *  w i iff A  w i – So µ[i, i, A] = P(A  w i )

39. CYK Base case “Can 1 you 2 book 3 ELAL 4 flights 5 ?” Aux 1 1.4 Noun 5 5.5 ……

40. CYK Recursive case  For strings of words of length > 1, A *  w ij iff there is at least one rule A  BC where B derives the first k words (between i and i-1 +k ) and C derives the remaining ones (between i+k and j)  (for each non-terminal) Choose the max among all possibilities A CB ii-1+ki+kj µ [i, j, A)] = µ [i, i-1 +k, B] * µ [i+k, j, C] * P(A  BC)

41. CYK Termination The max prob parse will be µ [1, n, S] w1,1w1,2w1,3w1,4w1,5 w2,2w2,3w2,4w2,5 w3,3w3,4w3,5 w4,4w4,5 w5,5 S

42. Top down: Early algorithm  Finds constituents and partial constituents in input  A  B C. D E is partial: only the first half of the A A BCDE A  B C. D E D + = A BCDE A  B C D. E i j i k jk

43. Early algorithm  Proceeds incrementally, left-to-right  Before it reads word 5, it has already built all hypotheses that are consistent with first 4 words  Reads word 5 & attaches it to immediately preceding hypotheses. Might yield new constituents that are then attached to hypotheses immediately preceding them …  Use a parse table as we did in CKY, so we can look up anything we’ve discovered so far. “Dynamic programming.”

44. Example (grammar) ROOT  S S  NP VPNP  Papa NP  Det NN  caviar NP  NP PPN  spoon VP  VP PPV  ate VP  V NPP  with PP  P NPDet  the Det  a

45. 0 0 ROOT. S initialize Remember this stands for (0, ROOT . S)

46. 0 0 ROOT. S 0 S. NP VP predict the kind of S we are looking for Remember this stands for (0, S . NP VP)

47. 0 0 ROOT. S 0 S. NP VP 0 NP. Det N 0 NP. NP PP 0 NP. Papa predict the kind of NP we are looking for (actually we’ll look for 3 kinds: any of the 3 will do)

48. 0 0 ROOT. S 0 S. NP VP 0 NP. Det N 0 NP. NP PP 0 NP. Papa 0 Det. the 0 Det. a predict the kind of Det we are looking for (2 kinds)

49. 0 0 ROOT. S 0 S. NP VP 0 NP. Det N 0 NP. NP PP 0 NP. Papa 0 Det. the 0 Det. a predict the kind of NP we’re looking for but we were already looking for these so don’t add duplicate goals! Note that this happened when we were processing a left-recursive rule.

50. 0 Papa 1 0 ROOT. S0 NP Papa. 0 S. NP VP 0 NP. Det N 0 NP. NP PP 0 NP. Papa 0 Det. the 0 Det. a scan: the desired word is in the input!

51. 0 Papa 1 0 ROOT. S0 NP Papa. 0 S. NP VP 0 NP. Det N 0 NP. NP PP 0 NP. Papa 0 Det. the 0 Det. a scan: failure

52. 0 Papa 1 0 ROOT. S0 NP Papa. 0 S. NP VP 0 NP. Det N 0 NP. NP PP 0 NP. Papa 0 Det. the 0 Det. a scan: failure

53. 0 Papa 1 0 ROOT. S0 NP Papa. 0 S. NP VP0 S NP. VP 0 NP. Det N0 NP NP. PP 0 NP. NP PP 0 NP. Papa 0 Det. the 0 Det. a attach the newly created NP (which starts at 0) to its customers (incomplete constituents that end at 0 and have NP after the dot)

54. 0 Papa 1 0 ROOT. S0 NP Papa. 0 S. NP VP0 S NP. VP 0 NP. Det N0 NP NP. PP 0 NP. NP PP1 VP. V NP 0 NP. Papa1 VP. VP PP 0 Det. the 0 Det. a predict

55. 0 Papa 1 0 ROOT. S0 NP Papa. 0 S. NP VP0 S NP. VP 0 NP. Det N0 NP NP. PP 0 NP. NP PP1 VP. V NP 0 NP. Papa1 VP. VP PP 0 Det. the1 PP. P NP 0 Det. a predict

56. 0 Papa 1 0 ROOT. S0 NP Papa. 0 S. NP VP0 S NP. VP 0 NP. Det N0 NP NP. PP 0 NP. NP PP1 VP. V NP 0 NP. Papa1 VP. VP PP 0 Det. the1 PP. P NP 0 Det. a1 V. ate predict

57. 0 Papa 1 0 ROOT. S0 NP Papa. 0 S. NP VP0 S NP. VP 0 NP. Det N0 NP NP. PP 0 NP. NP PP1 VP. V NP 0 NP. Papa1 VP. VP PP 0 Det. the1 PP. P NP 0 Det. a1 V. ate predict

58. 0 Papa 1 0 ROOT. S0 NP Papa. 0 S. NP VP0 S NP. VP 0 NP. Det N0 NP NP. PP 0 NP. NP PP1 VP. V NP 0 NP. Papa1 VP. VP PP 0 Det. the1 PP. P NP 0 Det. a1 V. ate 1 P. with predict

59. 0 Papa 1 ate 2 0 ROOT. S0 NP Papa.1 V ate. 0 S. NP VP0 S NP. VP 0 NP. Det N0 NP NP. PP 0 NP. NP PP1 VP. V NP 0 NP. Papa1 VP. VP PP 0 Det. the1 PP. P NP 0 Det. a1 V. ate 1 P. with scan: success!

60. 0 Papa 1 ate 2 0 ROOT. S0 NP Papa.1 V ate. 0 S. NP VP0 S NP. VP 0 NP. Det N0 NP NP. PP 0 NP. NP PP1 VP. V NP 0 NP. Papa1 VP. VP PP 0 Det. the1 PP. P NP 0 Det. a1 V. ate 1 P. with scan: failure

61. 0 Papa 1 ate 2 0 ROOT. S0 NP Papa.1 V ate. 0 S. NP VP0 S NP. VP1 VP V. NP 0 NP. Det N0 NP NP. PP 0 NP. NP PP1 VP. V NP 0 NP. Papa1 VP. VP PP 0 Det. the1 PP. P NP 0 Det. a1 V. ate 1 P. with attach

62. 0 Papa 1 ate 2 0 ROOT. S0 NP Papa.1 V ate. 0 S. NP VP0 S NP. VP1 VP V. NP 0 NP. Det N0 NP NP. PP2 NP. Det N 0 NP. NP PP1 VP. V NP2 NP. NP PP 0 NP. Papa1 VP. VP PP2 NP. Papa 0 Det. the1 PP. P NP 0 Det. a1 V. ate 1 P. with predict

63. 0 Papa 1 ate 2 0 ROOT. S0 NP Papa.1 V ate. 0 S. NP VP0 S NP. VP1 VP V. NP 0 NP. Det N0 NP NP. PP2 NP. Det N 0 NP. NP PP1 VP. V NP2 NP. NP PP 0 NP. Papa1 VP. VP PP2 NP. Papa 0 Det. the1 PP. P NP2 Det. the 0 Det. a1 V. ate2 Det. a 1 P. with predict (these next few steps should look familiar)

64. 0 Papa 1 ate 2 0 ROOT. S0 NP Papa.1 V ate. 0 S. NP VP0 S NP. VP1 VP V. NP 0 NP. Det N0 NP NP. PP2 NP. Det N 0 NP. NP PP1 VP. V NP2 NP. NP PP 0 NP. Papa1 VP. VP PP2 NP. Papa 0 Det. the1 PP. P NP2 Det. the 0 Det. a1 V. ate2 Det. a 1 P. with predict

65. 0 Papa 1 ate 2 0 ROOT. S0 NP Papa.1 V ate. 0 S. NP VP0 S NP. VP1 VP V. NP 0 NP. Det N0 NP NP. PP2 NP. Det N 0 NP. NP PP1 VP. V NP2 NP. NP PP 0 NP. Papa1 VP. VP PP2 NP. Papa 0 Det. the1 PP. P NP2 Det. the 0 Det. a1 V. ate2 Det. a 1 P. with scan (this time we fail since Papa is not the next word)

66. 0 Papa 1 ate 2 the 3 0 ROOT. S0 NP Papa.1 V ate.2 Det the. 0 S. NP VP0 S NP. VP1 VP V. NP 0 NP. Det N0 NP NP. PP2 NP. Det N 0 NP. NP PP1 VP. V NP2 NP. NP PP 0 NP. Papa1 VP. VP PP2 NP. Papa 0 Det. the1 PP. P NP2 Det. the 0 Det. a1 V. ate2 Det. a 1 P. with scan: success!

67. 0 Papa 1 ate 2 the 3 0 ROOT. S0 NP Papa.1 V ate.2 Det the. 0 S. NP VP0 S NP. VP1 VP V. NP 0 NP. Det N0 NP NP. PP2 NP. Det N 0 NP. NP PP1 VP. V NP2 NP. NP PP 0 NP. Papa1 VP. VP PP2 NP. Papa 0 Det. the1 PP. P NP2 Det. the 0 Det. a1 V. ate2 Det. a 1 P. with

68. 0 Papa 1 ate 2 the 3 0 ROOT. S0 NP Papa.1 V ate.2 Det the. 0 S. NP VP0 S NP. VP1 VP V. NP2 NP Det. N 0 NP. Det N0 NP NP. PP2 NP. Det N 0 NP. NP PP1 VP. V NP2 NP. NP PP 0 NP. Papa1 VP. VP PP2 NP. Papa 0 Det. the1 PP. P NP2 Det. the 0 Det. a1 V. ate2 Det. a 1 P. with

69. 0 Papa 1 ate 2 the 3 0 ROOT. S0 NP Papa.1 V ate.2 Det the. 0 S. NP VP0 S NP. VP1 VP V. NP2 NP Det. N 0 NP. Det N0 NP NP. PP2 NP. Det N3 N. caviar 0 NP. NP PP1 VP. V NP2 NP. NP PP3 N. spoon 0 NP. Papa1 VP. VP PP2 NP. Papa 0 Det. the1 PP. P NP2 Det. the 0 Det. a1 V. ate2 Det. a 1 P. with

70. 0 Papa 1 ate 2 the 3 caviar 4 0 ROOT. S0 NP Papa.1 V ate.2 Det the.3 N caviar. 0 S. NP VP0 S NP. VP1 VP V. NP2 NP Det. N 0 NP. Det N0 NP NP. PP2 NP. Det N3 N. caviar 0 NP. NP PP1 VP. V NP2 NP. NP PP3 N. spoon 0 NP. Papa1 VP. VP PP2 NP. Papa 0 Det. the1 PP. P NP2 Det. the 0 Det. a1 V. ate2 Det. a 1 P. with

71. 0 Papa 1 ate 2 the 3 caviar 4 0 ROOT. S0 NP Papa.1 V ate.2 Det the.3 N caviar. 0 S. NP VP0 S NP. VP1 VP V. NP2 NP Det. N 0 NP. Det N0 NP NP. PP2 NP. Det N3 N. caviar 0 NP. NP PP1 VP. V NP2 NP. NP PP3 N. spoon 0 NP. Papa1 VP. VP PP2 NP. Papa 0 Det. the1 PP. P NP2 Det. the 0 Det. a1 V. ate2 Det. a 1 P. with

72. 0 Papa 1 ate 2 the 3 caviar 4 0 ROOT. S0 NP Papa.1 V ate.2 Det the.3 N caviar. 0 S. NP VP0 S NP. VP1 VP V. NP2 NP Det. N2 NP Det N. 0 NP. Det N0 NP NP. PP2 NP. Det N3 N. caviar 0 NP. NP PP1 VP. V NP2 NP. NP PP3 N. spoon 0 NP. Papa1 VP. VP PP2 NP. Papa 0 Det. the1 PP. P NP2 Det. the 0 Det. a1 V. ate2 Det. a 1 P. with attach

73. 0 Papa 1 ate 2 the 3 caviar 4 0 ROOT. S0 NP Papa.1 V ate.2 Det the.3 N caviar. 0 S. NP VP0 S NP. VP1 VP V. NP2 NP Det. N2 NP Det N. 0 NP. Det N0 NP NP. PP2 NP. Det N3 N. caviar1 VP V NP. 0 NP. NP PP1 VP. V NP2 NP. NP PP3 N. spoon2 NP NP. PP 0 NP. Papa1 VP. VP PP2 NP. Papa 0 Det. the1 PP. P NP2 Det. the 0 Det. a1 V. ate2 Det. a 1 P. with attach (again!)

74. 0 Papa 1 ate 2 the 3 caviar 4 0 ROOT. S0 NP Papa.1 V ate.2 Det the.3 N caviar. 0 S. NP VP0 S NP. VP1 VP V. NP2 NP Det. N2 NP Det N. 0 NP. Det N0 NP NP. PP2 NP. Det N3 N. caviar1 VP V NP. 0 NP. NP PP1 VP. V NP2 NP. NP PP3 N. spoon2 NP NP. PP 0 NP. Papa1 VP. VP PP2 NP. Papa0 S NP VP. 0 Det. the1 PP. P NP2 Det. the1 VP VP. PP 0 Det. a1 V. ate2 Det. a 1 P. with attach (again!)

75. 0 Papa 1 ate 2 the 3 caviar 4 0 ROOT. S0 NP Papa.1 V ate.2 Det the.3 N caviar. 0 S. NP VP0 S NP. VP1 VP V. NP2 NP Det. N2 NP Det N. 0 NP. Det N0 NP NP. PP2 NP. Det N3 N. caviar1 VP V NP. 0 NP. NP PP1 VP. V NP2 NP. NP PP3 N. spoon2 NP NP. PP 0 NP. Papa1 VP. VP PP2 NP. Papa0 S NP VP. 0 Det. the1 PP. P NP2 Det. the1 VP VP. PP 0 Det. a1 V. ate2 Det. a4 PP. P NP 1 P. with

76. 0 Papa 1 ate 2 the 3 caviar 4 0 ROOT. S0 NP Papa.1 V ate.2 Det the.3 N caviar. 0 S. NP VP0 S NP. VP1 VP V. NP2 NP Det. N2 NP Det N. 0 NP. Det N0 NP NP. PP2 NP. Det N3 N. caviar1 VP V NP. 0 NP. NP PP1 VP. V NP2 NP. NP PP3 N. spoon2 NP NP. PP 0 NP. Papa1 VP. VP PP2 NP. Papa0 S NP VP. 0 Det. the1 PP. P NP2 Det. the1 VP VP. PP 0 Det. a1 V. ate2 Det. a4 PP. P NP 1 P. with0 ROOT S. attach (again!)

77. 0 Papa 1 ate 2 the 3 caviar 4 0 ROOT. S0 NP Papa.1 V ate.2 Det the.3 N caviar. 0 S. NP VP0 S NP. VP1 VP V. NP2 NP Det. N2 NP Det N. 0 NP. Det N0 NP NP. PP2 NP. Det N3 N. caviar1 VP V NP. 0 NP. NP PP1 VP. V NP2 NP. NP PP3 N. spoon2 NP NP. PP 0 NP. Papa1 VP. VP PP2 NP. Papa0 S NP VP. 0 Det. the1 PP. P NP2 Det. the1 VP VP. PP 0 Det. a1 V. ate2 Det. a4 PP. P NP 1 P. with0 ROOT S.

78. 0 Papa 1 ate 2 the 3 caviar 4 0 ROOT. S0 NP Papa.1 V ate.2 Det the.3 N caviar. 0 S. NP VP0 S NP. VP1 VP V. NP2 NP Det. N2 NP Det N. 0 NP. Det N0 NP NP. PP2 NP. Det N3 N. caviar1 VP V NP. 0 NP. NP PP1 VP. V NP2 NP. NP PP3 N. spoon2 NP NP. PP 0 NP. Papa1 VP. VP PP2 NP. Papa0 S NP VP. 0 Det. the1 PP. P NP2 Det. the1 VP VP. PP 0 Det. a1 V. ate2 Det. a4 PP. P NP 1 P. with0 ROOT S. 4 P. with

79. 0 Papa 1 ate 2 the 3 caviar 4 0 ROOT. S0 NP Papa.1 V ate.2 Det the.3 N caviar. 0 S. NP VP0 S NP. VP1 VP V. NP2 NP Det. N2 NP Det N. 0 NP. Det N0 NP NP. PP2 NP. Det N3 N. caviar1 VP V NP. 0 NP. NP PP1 VP. V NP2 NP. NP PP3 N. spoon2 NP NP. PP 0 NP. Papa1 VP. VP PP2 NP. Papa0 S NP VP. 0 Det. the1 PP. P NP2 Det. the1 VP VP. PP 0 Det. a1 V. ate2 Det. a4 PP. P NP 1 P. with0 ROOT S. 4 P. with

80. 0 Papa 1 ate 2 the 3 caviar 4 with 5 0 ROOT. S0 NP Papa.1 V ate.2 Det the.3 N caviar.4 P with. 0 S. NP VP0 S NP. VP1 VP V. NP2 NP Det. N2 NP Det N. 0 NP. Det N0 NP NP. PP2 NP. Det N3 N. caviar1 VP V NP. 0 NP. NP PP1 VP. V NP2 NP. NP PP3 N. spoon2 NP NP. PP 0 NP. Papa1 VP. VP PP2 NP. Papa0 S NP VP. 0 Det. the1 PP. P NP2 Det. the1 VP VP. PP 0 Det. a1 V. ate2 Det. a4 PP. P NP 1 P. with0 ROOT S. 4 P. with

81. 0 Papa 1 ate 2 the 3 caviar 4 with 5 0 ROOT. S0 NP Papa.1 V ate.2 Det the.3 N caviar.4 P with. 0 S. NP VP0 S NP. VP1 VP V. NP2 NP Det. N2 NP Det N.4 PP P. NP 0 NP. Det N0 NP NP. PP2 NP. Det N3 N. caviar1 VP V NP. 0 NP. NP PP1 VP. V NP2 NP. NP PP3 N. spoon2 NP NP. PP 0 NP. Papa1 VP. VP PP2 NP. Papa0 S NP VP. 0 Det. the1 PP. P NP2 Det. the1 VP VP. PP 0 Det. a1 V. ate2 Det. a4 PP. P NP 1 P. with0 ROOT S. 4 P. with

82. 0 Papa 1 ate 2 the 3 caviar 4 with 5 0 ROOT. S0 NP Papa.1 V ate.2 Det the.3 N caviar.4 P with. 0 S. NP VP0 S NP. VP1 VP V. NP2 NP Det. N2 NP Det N.4 PP P. NP 0 NP. Det N0 NP NP. PP2 NP. Det N3 N. caviar1 VP V NP.5 NP. Det N 0 NP. NP PP1 VP. V NP2 NP. NP PP3 N. spoon2 NP NP. PP5 NP. NP PP 0 NP. Papa1 VP. VP PP2 NP. Papa0 S NP VP.5 NP. Papa 0 Det. the1 PP. P NP2 Det. the1 VP VP. PP 0 Det. a1 V. ate2 Det. a4 PP. P NP 1 P. with0 ROOT S. 4 P. with

83. 0 Papa 1 ate 2 the 3 caviar 4 with 5 0 ROOT. S0 NP Papa.1 V ate.2 Det the.3 N caviar.4 P with. 0 S. NP VP0 S NP. VP1 VP V. NP2 NP Det. N2 NP Det N.4 PP P. NP 0 NP. Det N0 NP NP. PP2 NP. Det N3 N. caviar1 VP V NP.5 NP. Det N 0 NP. NP PP1 VP. V NP2 NP. NP PP3 N. spoon2 NP NP. PP5 NP. NP PP 0 NP. Papa1 VP. VP PP2 NP. Papa0 S NP VP.5 NP. Papa 0 Det. the1 PP. P NP2 Det. the1 VP VP. PP5 Det. the 0 Det. a1 V. ate2 Det. a4 PP. P NP5 Det. a 1 P. with0 ROOT S. 4 P. with

84. 0 Papa 1 ate 2 the 3 caviar 4 with 5 0 ROOT. S0 NP Papa.1 V ate.2 Det the.3 N caviar.4 P with. 0 S. NP VP0 S NP. VP1 VP V. NP2 NP Det. N2 NP Det N.4 PP P. NP 0 NP. Det N0 NP NP. PP2 NP. Det N3 N. caviar1 VP V NP.5 NP. Det N 0 NP. NP PP1 VP. V NP2 NP. NP PP3 N. spoon2 NP NP. PP5 NP. NP PP 0 NP. Papa1 VP. VP PP2 NP. Papa0 S NP VP.5 NP. Papa 0 Det. the1 PP. P NP2 Det. the1 VP VP. PP5 Det. the 0 Det. a1 V. ate2 Det. a4 PP. P NP5 Det. a 1 P. with0 ROOT S. 4 P. with

85. 0 Papa 1 ate 2 the 3 caviar 4 with 5 0 ROOT. S0 NP Papa.1 V ate.2 Det the.3 N caviar.4 P with. 0 S. NP VP0 S NP. VP1 VP V. NP2 NP Det. N2 NP Det N.4 PP P. NP 0 NP. Det N0 NP NP. PP2 NP. Det N3 N. caviar1 VP V NP.5 NP. Det N 0 NP. NP PP1 VP. V NP2 NP. NP PP3 N. spoon2 NP NP. PP5 NP. NP PP 0 NP. Papa1 VP. VP PP2 NP. Papa0 S NP VP.5 NP. Papa 0 Det. the1 PP. P NP2 Det. the1 VP VP. PP5 Det. the 0 Det. a1 V. ate2 Det. a4 PP. P NP5 Det. a 1 P. with0 ROOT S. 4 P. with

86. 0 Papa 1 ate 2 the 3 caviar 4 with 5 0 ROOT. S0 NP Papa.1 V ate.2 Det the.3 N caviar.4 P with. 0 S. NP VP0 S NP. VP1 VP V. NP2 NP Det. N2 NP Det N.4 PP P. NP 0 NP. Det N0 NP NP. PP2 NP. Det N3 N. caviar1 VP V NP.5 NP. Det N 0 NP. NP PP1 VP. V NP2 NP. NP PP3 N. spoon2 NP NP. PP5 NP. NP PP 0 NP. Papa1 VP. VP PP2 NP. Papa0 S NP VP.5 NP. Papa 0 Det. the1 PP. P NP2 Det. the1 VP VP. PP5 Det. the 0 Det. a1 V. ate2 Det. a4 PP. P NP5 Det. a 1 P. with0 ROOT S. 4 P. with

87. 0 Papa 1 ate 2 the 3 caviar 4 with 5 a 6 0 ROOT. S0 NP Papa.1 V ate.2 Det the.3 N caviar.4 P with.5 Det a. 0 S. NP VP0 S NP. VP1 VP V. NP2 NP Det. N2 NP Det N.4 PP P. NP 0 NP. Det N0 NP NP. PP2 NP. Det N3 N. caviar1 VP V NP.5 NP. Det N 0 NP. NP PP1 VP. V NP2 NP. NP PP3 N. spoon2 NP NP. PP5 NP. NP PP 0 NP. Papa1 VP. VP PP2 NP. Papa0 S NP VP.5 NP. Papa 0 Det. the1 PP. P NP2 Det. the1 VP VP. PP5 Det. the 0 Det. a1 V. ate2 Det. a4 PP. P NP5 Det. a 1 P. with0 ROOT S. 4 P. with

88. 0 Papa 1 ate 2 the 3 caviar 4 with 5 a 6 0 ROOT. S0 NP Papa.1 V ate.2 Det the.3 N caviar.4 P with.5 Det a. 0 S. NP VP0 S NP. VP1 VP V. NP2 NP Det. N2 NP Det N.4 PP P. NP5 NP Det. N 0 NP. Det N0 NP NP. PP2 NP. Det N3 N. caviar1 VP V NP.5 NP. Det N 0 NP. NP PP1 VP. V NP2 NP. NP PP3 N. spoon2 NP NP. PP5 NP. NP PP 0 NP. Papa1 VP. VP PP2 NP. Papa0 S NP VP.5 NP. Papa 0 Det. the1 PP. P NP2 Det. the1 VP VP. PP5 Det. the 0 Det. a1 V. ate2 Det. a4 PP. P NP5 Det. a 1 P. with0 ROOT S. 4 P. with

89. 0 Papa 1 ate 2 the 3 caviar 4 with 5 a 6 0 ROOT. S0 NP Papa.1 V ate.2 Det the.3 N caviar.4 P with.5 Det a. 0 S. NP VP0 S NP. VP1 VP V. NP2 NP Det. N2 NP Det N.4 PP P. NP5 NP Det. N 0 NP. Det N0 NP NP. PP2 NP. Det N3 N. caviar1 VP V NP.5 NP. Det N6 N. caviar 0 NP. NP PP1 VP. V NP2 NP. NP PP3 N. spoon2 NP NP. PP5 NP. NP PP6 N. spoon 0 NP. Papa1 VP. VP PP2 NP. Papa0 S NP VP.5 NP. Papa 0 Det. the1 PP. P NP2 Det. the1 VP VP. PP5 Det. the 0 Det. a1 V. ate2 Det. a4 PP. P NP5 Det. a 1 P. with0 ROOT S. 4 P. with

90. 0 Papa 1 ate 2 the 3 caviar 4 with 5 a 6 0 ROOT. S0 NP Papa.1 V ate.2 Det the.3 N caviar.4 P with.5 Det a. 0 S. NP VP0 S NP. VP1 VP V. NP2 NP Det. N2 NP Det N.4 PP P. NP5 NP Det. N 0 NP. Det N0 NP NP. PP2 NP. Det N3 N. caviar1 VP V NP.5 NP. Det N6 N. caviar 0 NP. NP PP1 VP. V NP2 NP. NP PP3 N. spoon2 NP NP. PP5 NP. NP PP6 N. spoon 0 NP. Papa1 VP. VP PP2 NP. Papa0 S NP VP.5 NP. Papa 0 Det. the1 PP. P NP2 Det. the1 VP VP. PP5 Det. the 0 Det. a1 V. ate2 Det. a4 PP. P NP5 Det. a 1 P. with0 ROOT S. 4 P. with

91. 0 Papa 1 ate 2 the 3 caviar 4 with 5 a 6 spoon 7 0 ROOT. S0 NP Papa.1 V ate.2 Det the.3 N caviar.4 P with.5 Det a.6 N spoon. 0 S. NP VP0 S NP. VP1 VP V. NP2 NP Det. N2 NP Det N.4 PP P. NP5 NP Det. N 0 NP. Det N0 NP NP. PP2 NP. Det N3 N. caviar1 VP V NP.5 NP. Det N6 N. caviar 0 NP. NP PP1 VP. V NP2 NP. NP PP3 N. spoon2 NP NP. PP5 NP. NP PP6 N. spoon 0 NP. Papa1 VP. VP PP2 NP. Papa0 S NP VP.5 NP. Papa 0 Det. the1 PP. P NP2 Det. the1 VP VP. PP5 Det. the 0 Det. a1 V. ate2 Det. a4 PP. P NP5 Det. a 1 P. with0 ROOT S. 4 P. with

92. 0 Papa 1 ate 2 the 3 caviar 4 with 5 a 6 spoon 7 0 ROOT. S0 NP Papa.1 V ate.2 Det the.3 N caviar.4 P with.5 Det a.6 N spoon. 0 S. NP VP0 S NP. VP1 VP V. NP2 NP Det. N2 NP Det N.4 PP P. NP5 NP Det. N5 NP Det N. 0 NP. Det N0 NP NP. PP2 NP. Det N3 N. caviar1 VP V NP.5 NP. Det N6 N. caviar 0 NP. NP PP1 VP. V NP2 NP. NP PP3 N. spoon2 NP NP. PP5 NP. NP PP6 N. spoon 0 NP. Papa1 VP. VP PP2 NP. Papa0 S NP VP.5 NP. Papa 0 Det. the1 PP. P NP2 Det. the1 VP VP. PP5 Det. the 0 Det. a1 V. ate2 Det. a4 PP. P NP5 Det. a 1 P. with0 ROOT S. 4 P. with

93. 0 Papa 1 ate 2 the 3 caviar 4 with 5 a 6 spoon 7 0 ROOT. S0 NP Papa.1 V ate.2 Det the.3 N caviar.4 P with.5 Det a.6 N spoon. 0 S. NP VP0 S NP. VP1 VP V. NP2 NP Det. N2 NP Det N.4 PP P. NP5 NP Det. N5 NP Det N. 0 NP. Det N0 NP NP. PP2 NP. Det N3 N. caviar1 VP V NP.5 NP. Det N6 N. caviar4 PP P NP. 0 NP. NP PP1 VP. V NP2 NP. NP PP3 N. spoon2 NP NP. PP5 NP. NP PP6 N. spoon5 NP NP. PP 0 NP. Papa1 VP. VP PP2 NP. Papa0 S NP VP.5 NP. Papa 0 Det. the1 PP. P NP2 Det. the1 VP VP. PP5 Det. the 0 Det. a1 V. ate2 Det. a4 PP. P NP5 Det. a 1 P. with0 ROOT S. 4 P. with

94. 0 Papa 1 ate 2 the 3 caviar 4 with a spoon 7 0 ROOT. S0 NP Papa.1 V ate.2 Det the.3 N caviar. … 6 N spoon. 0 S. NP VP0 S NP. VP1 VP V. NP2 NP Det. N2 NP Det N.5 NP Det N. 0 NP. Det N0 NP NP. PP2 NP. Det N3 N. caviar1 VP V NP.4 PP P NP. 0 NP. NP PP1 VP. V NP2 NP. NP PP3 N. spoon2 NP NP. PP5 NP NP. PP 0 NP. Papa1 VP. VP PP2 NP. Papa0 S NP VP.2 NP NP PP. 0 Det. the1 PP. P NP2 Det. the1 VP VP. PP1 VP VP PP. 0 Det. a1 V. ate2 Det. a4 PP. P NP 1 P. with0 ROOT S. 4 P. with

95. 0 Papa 1 ate 2 the 3 caviar 4 with a spoon 7 0 ROOT. S0 NP Papa.1 V ate.2 Det the.3 N caviar. … 6 N spoon. 0 S. NP VP0 S NP. VP1 VP V. NP2 NP Det. N2 NP Det N.5 NP Det N. 0 NP. Det N0 NP NP. PP2 NP. Det N3 N. caviar1 VP V NP.4 PP P NP. 0 NP. NP PP1 VP. V NP2 NP. NP PP3 N. spoon2 NP NP. PP5 NP NP. PP 0 NP. Papa1 VP. VP PP2 NP. Papa0 S NP VP.2 NP NP PP. 0 Det. the1 PP. P NP2 Det. the1 VP VP. PP1 VP VP PP. 0 Det. a1 V. ate2 Det. a4 PP. P NP7 PP. P NP 1 P. with0 ROOT S. 4 P. with

96. 0 Papa 1 ate 2 the 3 caviar 4 with a spoon 7 0 ROOT. S0 NP Papa.1 V ate.2 Det the.3 N caviar. … 6 N spoon. 0 S. NP VP0 S NP. VP1 VP V. NP2 NP Det. N2 NP Det N.5 NP Det N. 0 NP. Det N0 NP NP. PP2 NP. Det N3 N. caviar1 VP V NP.4 PP P NP. 0 NP. NP PP1 VP. V NP2 NP. NP PP3 N. spoon2 NP NP. PP5 NP NP. PP 0 NP. Papa1 VP. VP PP2 NP. Papa0 S NP VP.2 NP NP PP. 0 Det. the1 PP. P NP2 Det. the1 VP VP. PP1 VP VP PP. 0 Det. a1 V. ate2 Det. a4 PP. P NP7 PP. P NP 1 P. with0 ROOT S.1 VP V NP. 4 P. with2 NP NP. PP

97. 0 Papa 1 ate 2 the 3 caviar 4 with a spoon 7 0 ROOT. S0 NP Papa.1 V ate.2 Det the.3 N caviar. … 6 N spoon. 0 S. NP VP0 S NP. VP1 VP V. NP2 NP Det. N2 NP Det N.5 NP Det N. 0 NP. Det N0 NP NP. PP2 NP. Det N3 N. caviar1 VP V NP.4 PP P NP. 0 NP. NP PP1 VP. V NP2 NP. NP PP3 N. spoon2 NP NP. PP5 NP NP. PP 0 NP. Papa1 VP. VP PP2 NP. Papa0 S NP VP.2 NP NP PP. 0 Det. the1 PP. P NP2 Det. the1 VP VP. PP1 VP VP PP. 0 Det. a1 V. ate2 Det. a4 PP. P NP7 PP. P NP 1 P. with0 ROOT S.1 VP V NP. 4 P. with2 NP NP. PP 0 S NP VP. 1 VP VP. PP

98. 0 Papa 1 ate 2 the 3 caviar 4 with a spoon 7 0 ROOT. S0 NP Papa.1 V ate.2 Det the.3 N caviar. … 6 N spoon. 0 S. NP VP0 S NP. VP1 VP V. NP2 NP Det. N2 NP Det N.5 NP Det N. 0 NP. Det N0 NP NP. PP2 NP. Det N3 N. caviar1 VP V NP.4 PP P NP. 0 NP. NP PP1 VP. V NP2 NP. NP PP3 N. spoon2 NP NP. PP5 NP NP. PP 0 NP. Papa1 VP. VP PP2 NP. Papa0 S NP VP.2 NP NP PP. 0 Det. the1 PP. P NP2 Det. the1 VP VP. PP1 VP VP PP. 0 Det. a1 V. ate2 Det. a4 PP. P NP7 PP. P NP 1 P. with0 ROOT S.1 VP V NP. 4 P. with2 NP NP. PP 0 S NP VP. 1 VP VP. PP 7 P. with

99. 0 Papa 1 ate 2 the 3 caviar 4 with a spoon 7 0 ROOT. S0 NP Papa.1 V ate.2 Det the.3 N caviar. … 6 N spoon. 0 S. NP VP0 S NP. VP1 VP V. NP2 NP Det. N2 NP Det N.5 NP Det N. 0 NP. Det N0 NP NP. PP2 NP. Det N3 N. caviar1 VP V NP.4 PP P NP. 0 NP. NP PP1 VP. V NP2 NP. NP PP3 N. spoon2 NP NP. PP5 NP NP. PP 0 NP. Papa1 VP. VP PP2 NP. Papa0 S NP VP.2 NP NP PP. 0 Det. the1 PP. P NP2 Det. the1 VP VP. PP1 VP VP PP. 0 Det. a1 V. ate2 Det. a4 PP. P NP7 PP. P NP 1 P. with0 ROOT S.1 VP V NP. 4 P. with2 NP NP. PP 0 S NP VP. 1 VP VP. PP 7 P. with

100. 0 Papa 1 ate 2 the 3 caviar 4 with a spoon 7 0 ROOT. S0 NP Papa.1 V ate.2 Det the.3 N caviar. … 6 N spoon. 0 S. NP VP0 S NP. VP1 VP V. NP2 NP Det. N2 NP Det N.5 NP Det N. 0 NP. Det N0 NP NP. PP2 NP. Det N3 N. caviar1 VP V NP.4 PP P NP. 0 NP. NP PP1 VP. V NP2 NP. NP PP3 N. spoon2 NP NP. PP5 NP NP. PP 0 NP. Papa1 VP. VP PP2 NP. Papa0 S NP VP.2 NP NP PP. 0 Det. the1 PP. P NP2 Det. the1 VP VP. PP1 VP VP PP. 0 Det. a1 V. ate2 Det. a4 PP. P NP7 PP. P NP 1 P. with0 ROOT S.1 VP V NP. 4 P. with2 NP NP. PP 0 S NP VP. 1 VP VP. PP 7 P. with

101. 0 Papa 1 ate 2 the 3 caviar 4 with a spoon 7 0 ROOT. S0 NP Papa.1 V ate.2 Det the.3 N caviar. … 6 N spoon. 0 S. NP VP0 S NP. VP1 VP V. NP2 NP Det. N2 NP Det N.5 NP Det N. 0 NP. Det N0 NP NP. PP2 NP. Det N3 N. caviar1 VP V NP.4 PP P NP. 0 NP. NP PP1 VP. V NP2 NP. NP PP3 N. spoon2 NP NP. PP5 NP NP. PP 0 NP. Papa1 VP. VP PP2 NP. Papa0 S NP VP.2 NP NP PP. 0 Det. the1 PP. P NP2 Det. the1 VP VP. PP1 VP VP PP. 0 Det. a1 V. ate2 Det. a4 PP. P NP7 PP. P NP 1 P. with0 ROOT S.1 VP V NP. 4 P. with2 NP NP. PP 0 S NP VP. 1 VP VP. PP 7 P. with 0 ROOT S.

102. 0 Papa 1 ate 2 the 3 caviar 4 with a spoon 7 0 ROOT. S0 NP Papa.1 V ate.2 Det the.3 N caviar. … 6 N spoon. 0 S. NP VP0 S NP. VP1 VP V. NP2 NP Det. N2 NP Det N.5 NP Det N. 0 NP. Det N0 NP NP. PP2 NP. Det N3 N. caviar1 VP V NP.4 PP P NP. 0 NP. NP PP1 VP. V NP2 NP. NP PP3 N. spoon2 NP NP. PP5 NP NP. PP 0 NP. Papa1 VP. VP PP2 NP. Papa0 S NP VP.2 NP NP PP. 0 Det. the1 PP. P NP2 Det. the1 VP VP. PP1 VP VP PP. 0 Det. a1 V. ate2 Det. a4 PP. P NP7 PP. P NP 1 P. with0 ROOT S.1 VP V NP. 4 P. with2 NP NP. PP 0 S NP VP. 1 VP VP. PP 7 P. with 0 ROOT S.

103. 0 Papa 1 ate 2 the 3 caviar 4 with a spoon 7 0 ROOT. S0 NP Papa.1 V ate.2 Det the.3 N caviar. … 6 N spoon. 0 S. NP VP0 S NP. VP1 VP V. NP2 NP Det. N2 NP Det N.5 NP Det N. 0 NP. Det N0 NP NP. PP2 NP. Det N3 N. caviar1 VP V NP.4 PP P NP. 0 NP. NP PP1 VP. V NP2 NP. NP PP3 N. spoon2 NP NP. PP5 NP NP. PP 0 NP. Papa1 VP. VP PP2 NP. Papa0 S NP VP.2 NP NP PP. 0 Det. the1 PP. P NP2 Det. the1 VP VP. PP1 VP VP PP. 0 Det. a1 V. ate2 Det. a4 PP. P NP7 PP. P NP 1 P. with0 ROOT S.1 VP V NP. 4 P. with2 NP NP. PP 0 S NP VP. 1 VP VP. PP 7 P. with 0 ROOT S.

104. 0 Papa 1 ate 2 the 3 caviar 4 with a spoon 7 0 ROOT. S0 NP Papa.1 V ate.2 Det the.3 N caviar. … 6 N spoon. 0 S. NP VP0 S NP. VP1 VP V. NP2 NP Det. N2 NP Det N.5 NP Det N. 0 NP. Det N0 NP NP. PP2 NP. Det N3 N. caviar1 VP V NP.4 PP P NP. 0 NP. NP PP1 VP. V NP2 NP. NP PP3 N. spoon2 NP NP. PP5 NP NP. PP 0 NP. Papa1 VP. VP PP2 NP. Papa0 S NP VP.2 NP NP PP. 0 Det. the1 PP. P NP2 Det. the1 VP VP. PP1 VP VP PP. 0 Det. a1 V. ate2 Det. a4 PP. P NP7 PP. P NP 1 P. with0 ROOT S.1 VP V NP. 4 P. with2 NP NP. PP 0 S NP VP. 1 VP VP. PP 7 P. with 0 ROOT S.

105. 0 Papa 1 ate 2 the 3 caviar 4 with a spoon 7 0 ROOT. S0 NP Papa.1 V ate.2 Det the.3 N caviar. … 6 N spoon. 0 S. NP VP0 S NP. VP1 VP V. NP2 NP Det. N2 NP Det N.5 NP Det N. 0 NP. Det N0 NP NP. PP2 NP. Det N3 N. caviar1 VP V NP.4 PP P NP. 0 NP. NP PP1 VP. V NP2 NP. NP PP3 N. spoon2 NP NP. PP5 NP NP. PP 0 NP. Papa1 VP. VP PP2 NP. Papa0 S NP VP.2 NP NP PP. 0 Det. the1 PP. P NP2 Det. the1 VP VP. PP1 VP VP PP. 0 Det. a1 V. ate2 Det. a4 PP. P NP7 PP. P NP 1 P. with0 ROOT S.1 VP V NP. 4 P. with2 NP NP. PP 0 S NP VP. 1 VP VP. PP 7 P. with 0 ROOT S.