1. CPSC 503 Computational Linguistics
Lecture 7
Giuseppe Carenini
9/20/2018
CPSC503 Winter 2012

2. Today 29 Jan Finish Context Free Grammars / English Syntax Parsing
9/20/2018
CPSC503 Winter 2012

3. Knowledge-Formalisms Map (next three lectures)
State Machines (and prob. versions)
(Finite State Automata,Finite State Transducers, Markov Models)
Morphology
Syntax
Rule systems (and prob. versions)
(e.g., (Prob.) Context-Free Grammars)
Semantics
My Conceptual map - This is the master plan
Markov Models used for part-of-speech and dialog
Syntax is the study of formal relationship between words
How words are clustered into classes (that determine how they group and behave)
How they group with they neighbors into phrases
Pragmatics
Discourse and Dialogue
Logical formalisms
(First-Order Logics)
AI planners
9/20/2018
CPSC503 Winter 2012

4. Context Free Grammar (summary)
9/20/2018
CPSC503 Winter 2012

5. Problems with CFGs Agreement Subcategorization 9/20/2018
CPSC503 Winter 2012

6. Agreement
In English,
Determiners and nouns have to agree in number
Subjects and verbs have to agree in person and number
Many languages have agreement systems that are far more complex than this (e.g., gender).
9/20/2018
CPSC503 Winter 2012

7. Agreement This dog Those dogs This dog eats You have it Those dogs eat
*This dogs
*Those dog
*This dog eat
*You has it
*Those dogs eats
Number
person
9/20/2018
CPSC503 Winter 2012

8. Possible CFG Solution OLD Grammar NEW Grammar S -> NP VP
NP -> Det Nom
VP -> V NP …
SgS -> SgNP SgVP
PlS -> PlNp PlVP
SgNP -> SgDet SgNom
PlNP -> PlDet PlNom
PlVP -> PlV NP
SgVP3p ->SgV3p NP …
Sg = singular
Pl = plural
9/20/2018
CPSC503 Winter 2012

9. CFG Solution for Agreement
It works and stays within the power of CFGs
But it doesn’t scale all that well (explosion in the number of rules)
9/20/2018
CPSC503 Winter 2012

10. Subcategorization
Def. It expresses constraints that a predicate (verb here) places on the number and type of its arguments (see first table)
*John sneezed the book
*I prefer United has a flight
*Give with a flight
It turns out that verb subcategorization facts will provide a key element for semantic analysis
(determining who did what to who in an event).
9/20/2018
CPSC503 Winter 2012

11. Subcategorization Sneeze: John sneezed
Find: Please find [a flight to NY]NP
Give: Give [me]NP[a cheaper fare]NP
Help: Can you help [me]NP[with a flight]PP
Prefer: I prefer [to leave earlier]TO-VP
Told: I was told [United has a flight]S …
9/20/2018
CPSC503 Winter 2012

12. So? So the various rules for VPs overgenerate.
They allow strings containing verbs and arguments that don’t go together
For example:
VP -> V NP therefore Sneezed the book
VP -> V S therefore go she will go there
9/20/2018
CPSC503 Winter 2012

13. This solution has the same problem as the one for agreement
Possible CFG Solution
OLD Grammar
NEW Grammar
VP -> IntransV
VP -> TransV NP
VP -> TransPPto NP PPto …
TransPPto -> hand,give,..
VP -> V
VP -> V NP
VP -> V NP PP …
This solution has the same problem as the one for agreement
9/20/2018
CPSC503 Winter 2012

14. CFG for NLP: summary CFGs cover most syntactic structure in English.
But there are problems (over-generation)
That can be dealt with adequately, although not elegantly, by staying within the CFG framework.
Many practical computational grammars simply rely on CFG
For more elegant / concise approaches see Chpt 15 “Features and Unification”
CFGs appear to be just about what we need to account for a lot of basic syntactic structure in English.
But there are problems
That can be dealt with adequately, although not elegantly, by staying within the CFG framework.
There are simpler, more elegant, solutions that take us out of the CFG framework (beyond its formal power)
LFG, HPSG, Construction grammar, XTAG, etc.
Chapter 15 explores the unification approach in more detail
9/20/2018
CPSC503 Winter 2012

15. Dependency Grammars
Syntactic structure: binary relations between words
Links: grammatical function or very general semantic relation
Abstract away from word-order variations (simpler grammars)
Useful features in many NLP applications (for classification, summarization and NLG)
9/20/2018
CPSC503 Winter 2012

16. Today 29 Jan Finish Context Free Grammars / English Syntax Parsing
9/20/2018
CPSC503 Winter 2012

17. Parsing with CFGs Parser CFG Valid parse trees Sequence of words
flight
Nominal
I prefer a morning flight
Parser
CFG
Parsing with CFGs refers to the task of assigning correct trees to input strings
Correct here means a tree that covers all and only the elements of the input and has an S at the top
It doesn’t actually mean that the system can select the correct tree from among the possible trees
Assign valid trees: covers all and only the elements of the input and has an S at the top
9/20/2018
CPSC503 Winter 2012

18. Parsing as Search
CFG
Search space of possible parse trees
S -> NP VP
S -> Aux NP VP
NP -> Det Noun
VP -> Verb
Det -> a
Noun -> flight
Verb -> left, arrive
Aux -> do, does
defines
As with everything of interest, parsing involves a search which involves the making of choices
Search space defined by the grammar
Two constraints should guide the search: the grammar (must be a tree with an S at the top) and the input (must cover all the words in the input)
Parsing: find all trees that cover all and only the words in the input
9/20/2018
CPSC503 Winter 2012

19. Constraints on Search Parser CFG (search space) Sequence of words
Valid parse trees
flight
Nominal
I prefer a morning flight
Parser
CFG
(search space)
Parsing with CFGs refers to the task of assigning correct trees to input strings
Correct here means a tree that covers all and only the elements of the input and has an S at the top
It doesn’t actually mean that the system can select the correct tree from among the possible trees
Search Strategies:
Top-down or goal-directed
Bottom-up or data-directed
9/20/2018
CPSC503 Winter 2012

20. Top-Down Parsing
Since we’re trying to find trees rooted with an S (Sentences) start with the rules that give us an S.
Then work your way down from there to the words.
flight
Input:
We’ll start with some basic (meaning bad) methods before moving on to the one or two that you need to know
9/20/2018
CPSC503 Winter 2012

21. Next step: Top Down Space
……..
When POS categories are reached, reject trees whose leaves fail to match all words in the input
9/20/2018
CPSC503 Winter 2012

22. Bottom-Up Parsing
Of course, we also want trees that cover the input words. So start with trees that link up with the words in the right way.
Then work your way up from there.
flight
9/20/2018
CPSC503 Winter 2012

23. Two more steps: Bottom-Up Space
……..
flight
flight
flight
9/20/2018
CPSC503 Winter 2012

24. Top-Down vs. Bottom-Up Top-down Bottom-up
Only searches for trees that can be answers
But suggests trees that are not consistent with the words
Bottom-up
Only forms trees consistent with the words
Suggest trees that make no sense globally
9/20/2018
CPSC503 Winter 2012

25. So Combine Them Top-down: control strategy to generate trees
Bottom-up: to filter out inappropriate parses
Top-down Control strategy:
Depth vs. Breadth first
Which node to try to expand next
Which grammar rule to use to expand a node
(left-most)
There are a million ways to combine top-down expectations with bottom-up data to get more efficient searches
To incorporate input as soon as possible
3 choices to determine a control strategy
Natural forward incorporation of the input
(textual order)
9/20/2018
CPSC503 Winter 2012

26. Top-Down, Depth-First, Left-to-Right Search
Sample sentence: “Does this flight include a meal?”
Search state a partial parse tree and the next word to match in []
9/20/2018
CPSC503 Winter 2012

27. Example “Does this flight include a meal?” 9/20/2018
CPSC503 Winter 2012

28. Example “Does this flight include a meal?” flight flight 9/20/2018
CPSC503 Winter 2012

29. Example “Does this flight include a meal?” flight flight 9/20/2018
CPSC503 Winter 2012

30. Adding Bottom-up Filtering
The following sequence was a waste of time because an NP cannot generate a parse tree starting with an AUX
You should not bother expanding a node if it cannot generate a parse tree matching the next word
Aux
Aux
Aux
Aux
9/20/2018
CPSC503 Winter 2012

31. Bottom-Up Filtering Category Left Corners S
Det, Proper-Noun, Aux, Verb NP
Det, Proper-Noun
Nominal
Noun VP
Verb
Aux
9/20/2018
CPSC503 Winter 2012

32. Problems with TD-BU-filtering
Ambiguity
Repeated Parsing
The number of valid parses can grow exponentially in the number of phrases
(most of them do not make sense semantically so we do not consider them)
Parser builds valid trees for portions of the input, then discards them during backtracking, only to find out that it has to rebuild them again
SOLUTION: Earley Algorithm
(once again dynamic programming!)
9/20/2018
CPSC503 Winter 2012

33. (1) Structural Ambiguity
#of PP
# of NP parses … 6
429 7
1430
Three basic kinds:
Attachment/Coordination/NP-bracketing
“I shot an elephant in my pajamas”
What are other kinds of ambiguity?
VP -> V NP ; NP -> NP PP
VP -> V NP PP
Attachment non-PP “I saw Mary passing by cs2”
Coordination “new student and profs”
NP-bracketing “French language teacher”
In combinatorial mathematics, the Catalan numbers form a sequence of natural numbers that occur in various counting problems, often involving recursively defined objects.
Catalan numbers (2n)! / (n+1)! n!
9/20/2018
CPSC503 Winter 2012

34. (2) Repeated Work
Parsing is hard, and slow. It’s wasteful to redo stuff over and over and over.
Consider an attempt to top-down parse the following as an NP
“A flight from Indi to Houston on TWA”
9/20/2018
CPSC503 Winter 2012

35. starts from…. NP -> Det Nom NP-> NP PP Nom -> Noun ……
fails and backtracks
flight
9/20/2018
CPSC503 Winter 2012

36. restarts from…. NP -> Det Nom NP-> NP PP Nom -> Noun
fails and backtracks
flight
9/20/2018
CPSC503 Winter 2012

37. restarts from…. fails and backtracks.. flight 9/20/2018
CPSC503 Winter 2012

38. restarts from….
Success!
9/20/2018
CPSC503 Winter 2012

39. 4
But…. 3 2 1
9/20/2018
CPSC503 Winter 2012

40. Dynamic Programming Fills tables with solution to subproblems
Parsing: sub-trees consistent with the input, once discovered, are stored and can be reused
Stores ambiguous parse compactly
Does not do (avoidable) repeated work
Solves an exponential problem in polynomial time
We need a method that fills a “table” with partial results that
Sub-trees consistent with portions of the input…
9/20/2018
CPSC503 Winter 2012

41. Earley Parsing O(N 3)
Fills a table in a single sweep over the input words
Table is length N +1; N is number of words
Table entries represent:
Predicted constituents
In-progress constituents
Completed constituents and their locations
9/20/2018
CPSC503 Winter 2012

42. States The table-entries are called states and express:
what is predicted from that point
What is in progress at that point
what has been recognized up to that point
Representation: dotted-rules + location
S -> · VP [0,0] A VP is predicted at the start of the sentence
NP -> Det · Nominal [1,2] An NP is in progress; the Det goes from 1 to 2
VP -> V NP · [0,3] A VP has been found starting at 0 and ending at 3
Predicted / expected
9/20/2018
CPSC503 Winter 2012

43. Earley: answer S –> · [0,n]
Answer found by looking in the table (chart) in the right place.
The following state should be in the final column:
S –> · [0,n]
As with most dynamic programming approaches,
i.e., an S state that spans from 0 to n and is complete.
9/20/2018
CPSC503 Winter 2012

44. Earley Parsing Procedure
So sweep through the table from 0 to n in order, applying one of three operators to each state:
predictor: add top-down predictions to the chart
scanner: read input and add corresponding state to chart
completer: move dot to right when new constituent found
Results (new states) added to current or next set of states in chart
No backtracking and no states removed
9/20/2018
CPSC503 Winter 2012

45. Predictor Intuition: new states represent top-down expectations
Applied when non-part-of-speech non-terminals are to the right of a dot
S --> • VP [0,0]
Adds new states to end of current chart
One new state for each expansion of the non-terminal in the grammar
VP --> • V [0,0]
VP --> • V NP [0,0]
9/20/2018
CPSC503 Winter 2012

46. Scanner (part of speech)
New states for predicted part of speech.
Applicable when part of speech is to the right of a dot
VP --> • Verb NP [0,0] ( 0 “Book…” 1 )
Looks at current word in input
If match, adds state(s) to next chart
Verb --> book • [0,1]
9/20/2018
CPSC503 Winter 2012

47. Completer
Intuition: we’ve found a constituent, so tell everyone waiting for this
Applied when dot has reached right end of rule
NP --> Det Nom • [1,3]
Find all states w/dot at 1 and expecting an NP
VP --> V • NP [0,1]
Adds new (completed) state(s) to current chart
VP --> V NP • [0,3]
9/20/2018
CPSC503 Winter 2012

48. Example: “Book that flight”
We should find… an S from 0 to 3 that is a completed state…
9/20/2018
CPSC503 Winter 2012

49. Example
“Book that flight”
9/20/2018
CPSC503 Winter 2012

50. So far only a recognizer…
To generate all parses:
When old states waiting for the just completed constituent are updated => add a pointer from each “updated” to “completed”
Chart [0]
…..
S5 S->.VP [0,0] []
S6 VP -> . Verb [0,0] []
S7 VP -> . Verb NP [0,0] [] ….
Chart [1]
S8 Verb -> book . [0,1] []
S9 VP -> Verb . [0,1] [S8]
S10 S->VP. [0,1] [S9]
S11 VP->Verb . NP [0,1] [??] …. S8
Then simply read off all the backpointers from every complete S in the last column of the table
9/20/2018
CPSC503 Winter 2012

51. Error Handling
What happens when we look at the contents of the last table column and don't find a S -->  state?
Is it a total loss?
Chart contains every constituent and combination of constituents possible for the input given the grammar
Also useful for partial parsing or shallow parsing used in information extraction
9/20/2018
CPSC503 Winter 2012

52. A probabilistic earley parser as a psycholinguistic model
Author: John Hale The Johns Hopkins University, Baltimore MD Published in: · Proceeding NAACL '01 Proceedings of the second meeting of the North American Chapter of the Association for Computational Linguistics
· Citation Count: 27
9/20/2018
CPSC503 Winter 2012

53. Dynamic Programming Approaches
Earley
Top-down, no filtering, no restriction on grammar form
CKY (will see this applied to Prob. CFG)
Bottom-up, no filtering, grammars restricted to Chomsky-Normal Form (CNF) (i.e., -free and each production either A-> BC or A-> a)
9/20/2018
CPSC503 Winter 2012

54. For Next Time Read Chapter 14: Statistical Parsing
Optional: Read Chapter 15 (Features and Unification) – skip algorithms and implementation
9/20/2018
CPSC503 Winter 2012

55. Grammars and Constituency
Of course, there’s nothing easy or obvious about how we come up with right set of constituents and the rules that govern how they combine...
That’s why there are so many different theories of grammar and competing analyses of the same data.
The approach to grammar, and the analyses, adopted here are very generic (and don’t correspond to any modern linguistic theory of grammar).
9/20/2018
CPSC503 Winter 2012

56. Syntactic Notions so far...
N-grams: prob. distr. for next word can be effectively approximated knowing previous n words
POS categories are based on:
distributional properties (what other words can occur nearby)
morphological properties (affixes they take)
Syntactic in the sense that they extend beyond a single word out of context
By syntax I mean the kind of implicit knowledge of your native language that you had mastered by the time you were 3 or 4 years old without explicit instruction
Not the kind of stuff you were later taught in school.
9/20/2018
CPSC503 Winter 2012