1/22/2016CPSC503 Winter CPSC 503 Computational Linguistics Lecture 8 Giuseppe Carenini

1/22/2016CPSC503 Winter Knowledge-Formalisms Map Logical formalisms (First-Order Logics) Rule systems (and prob. versions) (e.g., (Prob.) Context-Free Grammars) State Machines (and prob. versions) (Finite State Automata,Finite State Transducers, Markov Models) Morphology Syntax Pragmatics Discourse and Dialogue Semantics AI planners

1/22/2016CPSC503 Winter Today 1/10 Finish CFG for Syntax of NL (problems) Parsing The Earley Algorithm Partial Parsing: Chuncking Dependency Grammars / Parsing

1/22/2016CPSC503 Winter Problems with CFGs Agreement Subcategorization

1/22/2016CPSC503 Winter Agreement In English, –Determiners and nouns have to agree in number –Subjects and verbs have to agree in person and number Many languages have agreement systems that are far more complex than this (e.g., gender).

1/22/2016CPSC503 Winter Agreement This dog Those dogs This dog eats You have it Those dogs eat *This dogs *Those dog *This dog eat *You has it *Those dogs eats

1/22/2016CPSC503 Winter Possible CFG Solution S -> NP VP NP -> Det Nom VP -> V NP … SgS -> SgNP SgVP PlS -> PlNp PlVP SgNP -> SgDet SgNom PlNP -> PlDet PlNom PlVP -> PlV NP SgVP ->SgV NP … Sg = singular Pl = plural OLD GrammarNEW Grammar

1/22/2016CPSC503 Winter CFG Solution for Agreement It works and stays within the power of CFGs But it doesn’t scale all that well (explosion in the number of rules)

1/22/2016CPSC503 Winter Subcategorization *John sneezed the book *I prefer United has a flight *Give with a flight Def. It expresses constraints that a predicate (verb here) places on the number and type of its arguments (see first table)

1/22/2016CPSC503 Winter Subcategorization Sneeze: John sneezed Find: Please find [a flight to NY] NP Give: Give [me] NP [a cheaper fare] NP Help: Can you help [me] NP [with a flight] PP Prefer: I prefer [to leave earlier] TO-VP Told: I was told [United has a flight] S …

1/22/2016CPSC503 Winter So? So the various rules for VPs overgenerate. –They allow strings containing verbs and arguments that don’t go together –For example: VP -> V NP therefore Sneezed the book VP -> V S therefore go she will go there

1/22/2016CPSC503 Winter Possible CFG Solution VP -> V VP -> V NP VP -> V NP PP … VP -> IntransV VP -> TransV NP VP -> TransPP to NP PP to … TransPP to -> hand,give,.. This solution has the same problem as the one for agreement OLD Grammar NEW Grammar

1/22/2016CPSC503 Winter CFG for NLP: summary CFGs cover most syntactic structure in English. But there are problems (overgeneration) –That can be dealt with adequately, although not elegantly, by staying within the CFG framework. There are simpler, more elegant, solutions that take us out of the CFG framework: LFG, XTAGS… see Chpt 15 “Features and Unification”

1/22/2016CPSC503 Winter Today 1/10 Finish CFG for Syntax of NL (problems) Parsing The Earley Algorithm Partial Parsing: Chuncking Dependency Grammars / Parsing

1/22/2016CPSC503 Winter Parsing with CFGs Assign valid trees: covers all and only the elements of the input and has an S at the top Parser I prefer a morning flight flight Nominal CFG Sequence of words Valid parse trees

1/22/2016CPSC503 Winter Parsing as Search S -> NP VP S -> Aux NP VP NP -> Det Noun VP -> Verb Det -> a Noun -> flight Verb -> left, arrive Aux -> do, does Search space of possible parse trees CFG defines Parsing : find all trees that cover all and only the words in the input

1/22/2016CPSC503 Winter Constraints on Search Parser I prefer a morning flight flight Nominal CFG (search space) Sequence of wordsValid parse trees Search Strategies: Top-down or goal-directed Bottom-up or data-directed

1/22/2016CPSC503 Winter Top-Down Parsing Since we’re trying to find trees rooted with an S (Sentences) start with the rules that give us an S. Then work your way down from there to the words. flight Input:

1/22/2016CPSC503 Winter Next step: Top Down Space When POS categories are reached, reject trees whose leaves fail to match all words in the input ……..

1/22/2016CPSC503 Winter Bottom-Up Parsing Of course, we also want trees that cover the input words. So start with trees that link up with the words in the right way. Then work your way up from there. flight

1/22/2016CPSC503 Winter Two more steps: Bottom-Up Space flight ……..

1/22/2016CPSC503 Winter Top-Down vs. Bottom-Up Top-down –Only searches for trees that can be answers –But suggests trees that are not consistent with the words Bottom-up –Only forms trees consistent with the words –Suggest trees that make no sense globally

1/22/2016CPSC503 Winter So Combine Them Top-down: control strategy to generate trees Bottom-up: to filter out inappropriate parses Top-down Control strategy: Depth vs. Breadth first Which node to try to expand next Which grammar rule to use to expand a node (left-most) (textual order)

1/22/2016CPSC503 Winter Top-Down, Depth-First, Left-to- Right Search Sample sentence: “Does this flight include a meal?”

1/22/2016CPSC503 Winter Example “Does this flight include a meal?”

1/22/2016CPSC503 Winter flight Example “Does this flight include a meal?”

1/22/2016CPSC503 Winter flight Example “Does this flight include a meal?”

1/22/2016CPSC503 Winter Adding Bottom-up Filtering The following sequence was a waste of time because an NP cannot generate a parse tree starting with an Aux Aux

1/22/2016CPSC503 Winter Bottom-Up Filtering CategoryLeft Corners SDet, Proper-Noun, Aux, Verb NPDet, Proper-Noun NominalNoun VPVerb Aux

1/22/2016CPSC503 Winter Problems with TD-BU-filtering Left recursion Ambiguity Repeated Parsing SOLUTION: Earley Algorithm (once again dynamic programming!)

1/22/2016CPSC503 Winter (1) Left-Recursion These rules appears in most English grammars S -> S and S VP -> VP PP NP -> NP PP

1/22/2016CPSC503 Winter (2) Structural Ambiguity “I shot an elephant in my pajamas” #of PP # of NP parses …… …… Three basic kinds: Attachment/Coordination/NP-bracketing

1/22/2016CPSC503 Winter (3) Repeated Work Parsing is hard, and slow. It’s wasteful to redo stuff over and over and over. Consider an attempt to top-down parse the following as an NP “A flight from Indi to Houston on TWA”

1/22/2016CPSC503 Winter flight NP -> Det Nom NP-> NP PP Nom -> Noun …… fails and backtracks starts from….

1/22/2016CPSC503 Winter flight NP -> Det Nom NP-> NP PP Nom -> Noun fails and backtracks restarts from….

1/22/2016CPSC503 Winter flight restarts from…. fails and backtracks..

1/22/2016CPSC503 Winter restarts from…. Success!

1/22/2016CPSC503 Winter But….

1/22/2016CPSC503 Winter Dynamic Programming Fills tables with solution to subproblems Parsing: sub-trees consistent with the input, once discovered, are stored and can be reused 1.Does not fall prey to left-recursion 2.Stores ambiguous parse compactly 3.Does not do (avoidable) repeated work

1/22/2016CPSC503 Winter Earley Parsing O(N 3 ) Fills a table in a single sweep over the input words –Table is length N +1; N is number of words –Table entries represent: Predicted constituents In-progress constituents Completed constituents and their locations

1/22/2016CPSC503 Winter For Next Time Read 12.7 Read in Chapter 13 (Parsing): , 13.5 Optional: Read Chapter 16 (Features and Unification) – skip algorithms and implementation

Final Project: Decision Two ways: Select and NLP task / problem or a technique used in NLP that truly interests you Tasks: summarization of ……, computing similarity between two terms/sentences (skim through the textbook) Techniques: extensions / variations / combinations of what we saw in class – Max Entropy Classifiers or MM, Dirichlet Multinomial Distributions 1/22/2016CPSC503 Winter

Final Project: goals (and hopefully contributions ) Improve on a proposed solution by using a possibly more effective technique or by combining multiple techniques Proposing a novel (minimally is OK) different solution. Apply a technique which has been used for nlp taskA to a different nlp taskB. Apply a technique to a different dataset or to a different language Proposing a different evaluation measure 1/22/2016CPSC503 Winter

Final Project: Examples / Ideas Look on the course WebPage 1/22/2016CPSC503 Winter

1/22/2016CPSC503 Winter Today 1/10 Finish CFG for Syntax of NL Parsing The Earley Algorithm Partial Parsing: Chuncking

1/22/2016CPSC503 Winter States The table-entries are called states and express: what is predicted from that point what has been recognized up to that point Representation: dotted-rules + location S -> · VP [0,0]A VP is predicted at the start of the sentence NP -> Det · Nominal[1,2]An NP is in progress; the Det goes from 1 to 2 VP -> V NP · [0,3]A VP has been found starting at 0 and ending at 3

1/22/2016CPSC503 Winter Graphically S -> · VP [0,0] NP -> Det · Nominal[1,2] VP -> V NP · [0,3]

1/22/2016CPSC503 Winter Earley: answer Answer found by looking in the table in the right place. The following state should be in the final column: i.e., an S state that spans from 0 to n and is complete. S –>  · [0,n]

1/22/2016CPSC503 Winter Earley Parsing Procedure So sweep through the table from 0 to n in order, applying one of three operators to each state: –predictor: add top-down predictions to the chart –scanner: read input and add corresponding state to chart –completer: move dot to right when new constituent found Results (new states) added to current or next set of states in chart No backtracking and no states removed

1/22/2016CPSC503 Winter Predictor Intuition: new states represent top- down expectations Applied when non-part-of-speech non- terminals are to the right of a dot S --> VP [0,0] Adds new states to end of current chart –One new state for each expansion of the non-terminal in the grammar VP --> V [0,0] VP --> V NP [0,0]

1/22/2016CPSC503 Winter Scanner (part of speech) New states for predicted part of speech. Applicable when part of speech is to the right of a dot VP --> Verb NP [0,0] ( 0 “Book…” 1 ) Looks at current word in input If match, adds state(s) to next chart Verb --> book NP [0,1]

1/22/2016CPSC503 Winter Completer Intuition: we’ve found a constituent, so tell everyone waiting for this Applied when dot has reached right end of rule NP --> Det Nom [1,3] Find all states w/dot at 1 and expecting an NP VP --> V NP [0,1] Adds new (completed) state(s) to current chart VP --> V NP [0,3]

1/22/2016CPSC503 Winter Example: “Book that flight” We should find… an S from 0 to 3 that is a completed state…

1/22/2016CPSC503 Winter Example “Book that flight”

1/22/2016CPSC503 Winter So far only a recognizer… Then simply read off all the backpointers from every complete S in the last column of the table To generate all parses : When old states waiting for the just completed constituent are updated => add a pointer from each “updated” to “completed” Chart [0] ….. S5 S->.VP[0,0] [] S6 VP ->. Verb[0,0] [] S7 VP ->. Verb NP[0,0] [] …. Chart [1] S8 Verb -> book. [0,1] [] S9 VP -> Verb. [0,1] [S8] S10 S->VP. [0,1] [S9] S11 VP->Verb. NP [0,1] [??] ….

1/22/2016CPSC503 Winter Error Handling What happens when we look at the contents of the last table column and don't find a S -->  state? –Is it a total loss? No... –Chart contains every constituent and combination of constituents possible for the input given the grammar Also useful for partial parsing or shallow parsing used in information extraction

1/22/2016CPSC503 Winter Earley and Left Recursion So Earley solves the left-recursion problem without having to alter the grammar or artificially limiting the search. –Never place a state into the chart that’s already there –Copy states before advancing them

1/22/2016CPSC503 Winter Earley and Left Recursion: 1 S -> NP VP NP -> NP PP The first rule predicts –S -> · NP VP [0,0]that adds –NP -> · NP PP [0,0] –stops there since adding any subsequent prediction would be fruitless

1/22/2016CPSC503 Winter Earley and Left Recursion: 2 When a state gets advanced make a copy and leave the original alone… Say we haveNP -> · NP PP [0,0] We find an NP from 0 to 2 so we create NP -> NP · PP [0,2] But we leave the original state as is

1/22/2016CPSC503 Winter Dynamic Programming Approaches Earley –Top-down, no filtering, no restriction on grammar form CKY –Bottom-up, no filtering, grammars restricted to Chomsky-Normal Form (CNF) (i.e.,  -free and each production either A-> BC or A-> a)

1/22/2016CPSC503 Winter Today 4/10 The Earley Algorithm Partial Parsing: Chunking

1/22/2016CPSC503 Winter Chunking Classify only basic non-recursive phrases (NP, VP, AP, PP) –Find non-overlapping chunks –Assign labels to chunks Chunk: typically includes headword and pre-head material [NP The HD box] that [NP you] [VP ordered] [PP from] [NP Shaw] [VP never arrived]

1/22/2016CPSC503 Winter Approaches to Chunking (1): Finite- State Rule-Based Set of hand-crafted rules (no recursion!) e.g., NP -> (Det) Noun* Noun Implemented as FSTs (unionized/deteminized/minimized) F-measure To build tree-like structures several FSTs can be combined [Abney ’96]

1/22/2016CPSC503 Winter Approaches to Chunking (1): Finite- State Rule-Based … several FSTs can be combined

1/22/2016CPSC503 Winter Approaches to Chunking (2): Machine Learning A case of sequential classification IOB tagging: (I) internal, (O) outside, (B) beginning Internal and Beginning for each chunk type => size of tagset (2n + 1) where n is the num of chunk types Find an annotated corpus Select feature set Select and train a classifier

1/22/2016CPSC503 Winter Context window approach Typical features: –Current / previous / following words –Current / previous / following POS –Previous chunks

1/22/2016CPSC503 Winter Context window approach Specific choice of machine learning approach does not seem to matter F-measure range Common causes of errors: –POS tagger inaccuracies –Inconsistencies in training corpus –Ambiguities involving conjunctions (e.g., “late arrivals and cancellations/departure are common in winter” )

1/22/2016CPSC503 Winter For Next Time Read Chapter 14 (Probabilistic CFG and Parsing)

1/22/2016CPSC503 Winter Dependency Grammars Syntactic structure: binary relations between words Links: grammatical function or very general semantic relation Abstract away from word-order variations (simpler grammars) Useful features in many NLP applications (for classification, summarization and NLG)