1. 1/22/2016CPSC503 Winter 20081 CPSC 503 Computational Linguistics Lecture 8 Giuseppe Carenini

2. 1/22/2016CPSC503 Winter 20082 Knowledge-Formalisms Map Logical formalisms (First-Order Logics) Rule systems (and prob. versions) (e.g., (Prob.) Context-Free Grammars) State Machines (and prob. versions) (Finite State Automata,Finite State Transducers, Markov Models) Morphology Syntax Pragmatics Discourse and Dialogue Semantics AI planners

3. 1/22/2016CPSC503 Winter 20083 Today 1/10 Finish CFG for Syntax of NL (problems) Parsing The Earley Algorithm Partial Parsing: Chuncking Dependency Grammars / Parsing

4. 1/22/2016CPSC503 Winter 20084 Problems with CFGs Agreement Subcategorization

5. 1/22/2016CPSC503 Winter 20085 Agreement In English, –Determiners and nouns have to agree in number –Subjects and verbs have to agree in person and number Many languages have agreement systems that are far more complex than this (e.g., gender).

6. 1/22/2016CPSC503 Winter 20086 Agreement This dog Those dogs This dog eats You have it Those dogs eat *This dogs *Those dog *This dog eat *You has it *Those dogs eats

7. 1/22/2016CPSC503 Winter 20087 Possible CFG Solution S -> NP VP NP -> Det Nom VP -> V NP … SgS -> SgNP SgVP PlS -> PlNp PlVP SgNP -> SgDet SgNom PlNP -> PlDet PlNom PlVP -> PlV NP SgVP ->SgV NP … Sg = singular Pl = plural OLD GrammarNEW Grammar

8. 1/22/2016CPSC503 Winter 20088 CFG Solution for Agreement It works and stays within the power of CFGs But it doesn’t scale all that well (explosion in the number of rules)

9. 1/22/2016CPSC503 Winter 20089 Subcategorization *John sneezed the book *I prefer United has a flight *Give with a flight Def. It expresses constraints that a predicate (verb here) places on the number and type of its arguments (see first table)

10. 1/22/2016CPSC503 Winter 200810 Subcategorization Sneeze: John sneezed Find: Please find [a flight to NY] NP Give: Give [me] NP [a cheaper fare] NP Help: Can you help [me] NP [with a flight] PP Prefer: I prefer [to leave earlier] TO-VP Told: I was told [United has a flight] S …

11. 1/22/2016CPSC503 Winter 200811 So? So the various rules for VPs overgenerate. –They allow strings containing verbs and arguments that don’t go together –For example: VP -> V NP therefore Sneezed the book VP -> V S therefore go she will go there

12. 1/22/2016CPSC503 Winter 200812 Possible CFG Solution VP -> V VP -> V NP VP -> V NP PP … VP -> IntransV VP -> TransV NP VP -> TransPP to NP PP to … TransPP to -> hand,give,.. This solution has the same problem as the one for agreement OLD Grammar NEW Grammar

13. 1/22/2016CPSC503 Winter 200813 CFG for NLP: summary CFGs cover most syntactic structure in English. But there are problems (overgeneration) –That can be dealt with adequately, although not elegantly, by staying within the CFG framework. There are simpler, more elegant, solutions that take us out of the CFG framework: LFG, XTAGS… see Chpt 15 “Features and Unification”

14. 1/22/2016CPSC503 Winter 200814 Today 1/10 Finish CFG for Syntax of NL (problems) Parsing The Earley Algorithm Partial Parsing: Chuncking Dependency Grammars / Parsing

15. 1/22/2016CPSC503 Winter 200815 Parsing with CFGs Assign valid trees: covers all and only the elements of the input and has an S at the top Parser I prefer a morning flight flight Nominal CFG Sequence of words Valid parse trees

16. 1/22/2016CPSC503 Winter 200816 Parsing as Search S -> NP VP S -> Aux NP VP NP -> Det Noun VP -> Verb Det -> a Noun -> flight Verb -> left, arrive Aux -> do, does Search space of possible parse trees CFG defines Parsing : find all trees that cover all and only the words in the input

17. 1/22/2016CPSC503 Winter 200817 Constraints on Search Parser I prefer a morning flight flight Nominal CFG (search space) Sequence of wordsValid parse trees Search Strategies: Top-down or goal-directed Bottom-up or data-directed

18. 1/22/2016CPSC503 Winter 200818 Top-Down Parsing Since we’re trying to find trees rooted with an S (Sentences) start with the rules that give us an S. Then work your way down from there to the words. flight Input:

19. 1/22/2016CPSC503 Winter 200819 Next step: Top Down Space When POS categories are reached, reject trees whose leaves fail to match all words in the input ……..

20. 1/22/2016CPSC503 Winter 200820 Bottom-Up Parsing Of course, we also want trees that cover the input words. So start with trees that link up with the words in the right way. Then work your way up from there. flight

21. 1/22/2016CPSC503 Winter 200821 Two more steps: Bottom-Up Space flight ……..

22. 1/22/2016CPSC503 Winter 200822 Top-Down vs. Bottom-Up Top-down –Only searches for trees that can be answers –But suggests trees that are not consistent with the words Bottom-up –Only forms trees consistent with the words –Suggest trees that make no sense globally

23. 1/22/2016CPSC503 Winter 200823 So Combine Them Top-down: control strategy to generate trees Bottom-up: to filter out inappropriate parses Top-down Control strategy: Depth vs. Breadth first Which node to try to expand next Which grammar rule to use to expand a node (left-most) (textual order)

24. 1/22/2016CPSC503 Winter 200824 Top-Down, Depth-First, Left-to- Right Search Sample sentence: “Does this flight include a meal?”

25. 1/22/2016CPSC503 Winter 200825 Example “Does this flight include a meal?”

26. 1/22/2016CPSC503 Winter 200826 flight Example “Does this flight include a meal?”

27. 1/22/2016CPSC503 Winter 200827 flight Example “Does this flight include a meal?”

28. 1/22/2016CPSC503 Winter 200828 Adding Bottom-up Filtering The following sequence was a waste of time because an NP cannot generate a parse tree starting with an Aux Aux

29. 1/22/2016CPSC503 Winter 200829 Bottom-Up Filtering CategoryLeft Corners SDet, Proper-Noun, Aux, Verb NPDet, Proper-Noun NominalNoun VPVerb Aux

30. 1/22/2016CPSC503 Winter 200830 Problems with TD-BU-filtering Left recursion Ambiguity Repeated Parsing SOLUTION: Earley Algorithm (once again dynamic programming!)

31. 1/22/2016CPSC503 Winter 200831 (1) Left-Recursion These rules appears in most English grammars S -> S and S VP -> VP PP NP -> NP PP

32. 1/22/2016CPSC503 Winter 200832 (2) Structural Ambiguity “I shot an elephant in my pajamas” #of PP # of NP parses …… 6469 71430 …… Three basic kinds: Attachment/Coordination/NP-bracketing

33. 1/22/2016CPSC503 Winter 200833 (3) Repeated Work Parsing is hard, and slow. It’s wasteful to redo stuff over and over and over. Consider an attempt to top-down parse the following as an NP “A flight from Indi to Houston on TWA”

34. 1/22/2016CPSC503 Winter 200834 flight NP -> Det Nom NP-> NP PP Nom -> Noun …… fails and backtracks starts from….

35. 1/22/2016CPSC503 Winter 200835 flight NP -> Det Nom NP-> NP PP Nom -> Noun fails and backtracks restarts from….

36. 1/22/2016CPSC503 Winter 200836 flight restarts from…. fails and backtracks..

37. 1/22/2016CPSC503 Winter 200837 restarts from…. Success!

38. 1/22/2016CPSC503 Winter 200838 4 3 2 1 But….

39. 1/22/2016CPSC503 Winter 200839 Dynamic Programming Fills tables with solution to subproblems Parsing: sub-trees consistent with the input, once discovered, are stored and can be reused 1.Does not fall prey to left-recursion 2.Stores ambiguous parse compactly 3.Does not do (avoidable) repeated work

40. 1/22/2016CPSC503 Winter 200840 Earley Parsing O(N 3 ) Fills a table in a single sweep over the input words –Table is length N +1; N is number of words –Table entries represent: Predicted constituents In-progress constituents Completed constituents and their locations

41. 1/22/2016CPSC503 Winter 200841 For Next Time Read 12.7 Read in Chapter 13 (Parsing): 13.4.2, 13.5 Optional: Read Chapter 16 (Features and Unification) – skip algorithms and implementation

42. Final Project: Decision Two ways: Select and NLP task / problem or a technique used in NLP that truly interests you Tasks: summarization of ……, computing similarity between two terms/sentences (skim through the textbook) Techniques: extensions / variations / combinations of what we saw in class – Max Entropy Classifiers or MM, Dirichlet Multinomial Distributions 1/22/2016CPSC503 Winter 200842

43. Final Project: goals (and hopefully contributions ) Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques Proposing a novel (minimally is OK) different solution. Apply a technique which has been used for nlp taskA to a different nlp taskB. Apply a technique to a different dataset or to a different language Proposing a different evaluation measure 1/22/2016CPSC503 Winter 200843

44. Final Project: Examples / Ideas Look on the course WebPage 1/22/2016CPSC503 Winter 200844

45. 1/22/2016CPSC503 Winter 200845 Today 1/10 Finish CFG for Syntax of NL Parsing The Earley Algorithm Partial Parsing: Chuncking

46. 1/22/2016CPSC503 Winter 200846 States The table-entries are called states and express: what is predicted from that point what has been recognized up to that point Representation: dotted-rules + location S -> · VP [0,0]A VP is predicted at the start of the sentence NP -> Det · Nominal[1,2]An NP is in progress; the Det goes from 1 to 2 VP -> V NP · [0,3]A VP has been found starting at 0 and ending at 3

47. 1/22/2016CPSC503 Winter 200847 Graphically S -> · VP [0,0] NP -> Det · Nominal[1,2] VP -> V NP · [0,3]

48. 1/22/2016CPSC503 Winter 200848 Earley: answer Answer found by looking in the table in the right place. The following state should be in the final column: i.e., an S state that spans from 0 to n and is complete. S –>  · [0,n]

49. 1/22/2016CPSC503 Winter 200849 Earley Parsing Procedure So sweep through the table from 0 to n in order, applying one of three operators to each state: –predictor: add top-down predictions to the chart –scanner: read input and add corresponding state to chart –completer: move dot to right when new constituent found Results (new states) added to current or next set of states in chart No backtracking and no states removed

50. 1/22/2016CPSC503 Winter 200850 Predictor Intuition: new states represent top- down expectations Applied when non-part-of-speech non- terminals are to the right of a dot S --> VP [0,0] Adds new states to end of current chart –One new state for each expansion of the non-terminal in the grammar VP --> V [0,0] VP --> V NP [0,0]

51. 1/22/2016CPSC503 Winter 200851 Scanner (part of speech) New states for predicted part of speech. Applicable when part of speech is to the right of a dot VP --> Verb NP [0,0] ( 0 “Book…” 1 ) Looks at current word in input If match, adds state(s) to next chart Verb --> book NP [0,1]

52. 1/22/2016CPSC503 Winter 200852 Completer Intuition: we’ve found a constituent, so tell everyone waiting for this Applied when dot has reached right end of rule NP --> Det Nom [1,3] Find all states w/dot at 1 and expecting an NP VP --> V NP [0,1] Adds new (completed) state(s) to current chart VP --> V NP [0,3]

53. 1/22/2016CPSC503 Winter 200853 Example: “Book that flight” We should find… an S from 0 to 3 that is a completed state…

54. 1/22/2016CPSC503 Winter 200854 Example “Book that flight”

55. 1/22/2016CPSC503 Winter 200855 So far only a recognizer… Then simply read off all the backpointers from every complete S in the last column of the table To generate all parses : When old states waiting for the just completed constituent are updated => add a pointer from each “updated” to “completed” Chart [0] ….. S5 S->.VP[0,0] [] S6 VP ->. Verb[0,0] [] S7 VP ->. Verb NP[0,0] [] …. Chart [1] S8 Verb -> book. [0,1] [] S9 VP -> Verb. [0,1] [S8] S10 S->VP. [0,1] [S9] S11 VP->Verb. NP [0,1] [??] ….

56. 1/22/2016CPSC503 Winter 200856 Error Handling What happens when we look at the contents of the last table column and don't find a S -->  state? –Is it a total loss? No... –Chart contains every constituent and combination of constituents possible for the input given the grammar Also useful for partial parsing or shallow parsing used in information extraction

57. 1/22/2016CPSC503 Winter 200857 Earley and Left Recursion So Earley solves the left-recursion problem without having to alter the grammar or artificially limiting the search. –Never place a state into the chart that’s already there –Copy states before advancing them

58. 1/22/2016CPSC503 Winter 200858 Earley and Left Recursion: 1 S -> NP VP NP -> NP PP The first rule predicts –S -> · NP VP [0,0]that adds –NP -> · NP PP [0,0] –stops there since adding any subsequent prediction would be fruitless

59. 1/22/2016CPSC503 Winter 200859 Earley and Left Recursion: 2 When a state gets advanced make a copy and leave the original alone… Say we haveNP -> · NP PP [0,0] We find an NP from 0 to 2 so we create NP -> NP · PP [0,2] But we leave the original state as is

60. 1/22/2016CPSC503 Winter 200860 Dynamic Programming Approaches Earley –Top-down, no filtering, no restriction on grammar form CKY –Bottom-up, no filtering, grammars restricted to Chomsky-Normal Form (CNF) (i.e.,  -free and each production either A-> BC or A-> a)

61. 1/22/2016CPSC503 Winter 200861 Today 4/10 The Earley Algorithm Partial Parsing: Chunking

62. 1/22/2016CPSC503 Winter 200862 Chunking Classify only basic non-recursive phrases (NP, VP, AP, PP) –Find non-overlapping chunks –Assign labels to chunks Chunk: typically includes headword and pre-head material [NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]

63. 1/22/2016CPSC503 Winter 200863 Approaches to Chunking (1): Finite- State Rule-Based Set of hand-crafted rules (no recursion!) e.g., NP -> (Det) Noun* Noun Implemented as FSTs (unionized/deteminized/minimized) F-measure 85-92 To build tree-like structures several FSTs can be combined [Abney ’96]

64. 1/22/2016CPSC503 Winter 200864 Approaches to Chunking (1): Finite- State Rule-Based … several FSTs can be combined

65. 1/22/2016CPSC503 Winter 200865 Approaches to Chunking (2): Machine Learning A case of sequential classification IOB tagging: (I) internal, (O) outside, (B) beginning Internal and Beginning for each chunk type => size of tagset (2n + 1) where n is the num of chunk types Find an annotated corpus Select feature set Select and train a classifier

66. 1/22/2016CPSC503 Winter 200866 Context window approach Typical features: –Current / previous / following words –Current / previous / following POS –Previous chunks

67. 1/22/2016CPSC503 Winter 200867 Context window approach Specific choice of machine learning approach does not seem to matter F-measure 92-94 range Common causes of errors: –POS tagger inaccuracies –Inconsistencies in training corpus –Ambiguities involving conjunctions (e.g., “late arrivals and cancellations/departure are common in winter” )

68. 1/22/2016CPSC503 Winter 200868 For Next Time Read Chapter 14 (Probabilistic CFG and Parsing)

69. 1/22/2016CPSC503 Winter 200869 Dependency Grammars Syntactic structure: binary relations between words Links: grammatical function or very general semantic relation Abstract away from word-order variations (simpler grammars) Useful features in many NLP applications (for classification, summarization and NLG)