1. CS626-460: Language Technology for the Web/Natural Language Processing
Pushpak Bhattacharyya
CSE Dept., IIT Bombay
Probabilistic parsing

2. (mainly using Manning and Schutz, FSNLP)
PCFG
(mainly using Manning and Schutz, FSNLP)

3. Language Models Probabilistic / Stochastic Context Free Grammars:
Simple probabilistic models capable of handling recursion
A CFG with probabilities attached to rules
Rule probabilities  how likely is it that a particular rewrite rule is used?
Why PCFGs?
Intuitive probabilistic models for tree-structured languages
Algorithms are extensions of HMM algorithms
Better than the n-gram model for language modeling.

4. Formal Definition of PCFG
A PCFG consists of
A set of terminals {wk}, k = 1,….,V
{wk} = { child, teddy, bear, played…}
A set of non-terminals {Ni}, i = 1,…,n
{Ni} = { NP, VP, DT…}
A designated start symbol N1
A set of rules {Ni  j}, where j is a sequence of terminals & non-terminals
NP  DT NN
A corresponding set of rule probabilities

5. Rule Probabilities Rule probabilities are such that
E.g., P( NP  DT NN) = 0.2
P( NP  NN) = 0.5
P( NP  NP PP) = 0.3
P( NP  DT NN) = 0.2
Means 20 % of the training data parses use the rule NP  DT NN

6. Probability of a sentence
Notation :
wab – subsequence wa….wb
Nj dominates wa….wb
or yield(Nj) = wa….wb Nj NP
wa……………..wb
the..sweet..teddy ..bear
Probability of a sentence = P(w1m)
Where t is a parse tree of the sentence
If t is a parse tree for the sentence w1m, this will be 1 !!

7. Assumptions of the PCFG model
Place invariance :
P(NP  DT NN) is same in locations 1 and 2
Context-free :
P(NP  DT NN | anything outside “The child”) = P(NP  DT NN)
Ancestor free : At 2,
P(NP  DT NN|its ancestor is VP) = P(NP DT NN) S NP VP 1
The child NP 2
The child

8. Example PCFG Rules & Probabilities
S  NP VP 1.0
NP  DT NN 0.5
NP  NNS 0.3
NP  NP PP 0.2
PP  P NP 1.0
VP  VP PP
VP  VBD NP 0.4
DT  the 1.0
NN  gunman 0.5
NN  building 0.5
VBD  sprayed 1.0
NNS  bullets 1.0

9. Example Parse t1` The gunman sprayed the building with bullets. S1.0
P (t1) = 1.0 * 0.5 * 1.0 * 0.5 * 0.6 * 0.4 * 1.0 * 0.5 * 1.0 * 0.5 * 1.0 * 1.0 * 0.3 * =
NP0.5
VP0.6
NN0.5
DT1.0
VP0.4
PP1.0
P1.0
NP0.3
NP0.5
The
gunman
VBD1.0
DT1.0
NN0.5
with
NNS1.0
sprayed
the
building
bullets

10. Another Parse t2 The gunman sprayed the building with bullets. S1.0
P (t2) = 1.0 * 0.5 * 1.0 * 0.5 * 0.4 * 1.0 * 0.2 * 0.5 * 1.0 * 0.5 * 1.0 * 1.0 * 0.3 * =
NP0.5
VP0.4
NN0.5
DT1.0
VBD1.0
NP0.2
The
gunman
sprayed
NP0.5
PP1.0
DT1.0
NN0.5
P1.0
NP0.3
NNS1.0
the
building
with
bullets

11. HMM ↔ PCFG O observed sequence ↔ w1m sentence
X state sequence ↔ t parse tree
 model ↔ G grammar
Three fundamental questions

12. HMM ↔ PCFG
How likely is a certain observation given the model? ↔ How likely is a sentence given the grammar?
How to choose a state sequence which best explains the observations? ↔ How to choose a parse which best supports the sentence? ↔ ↔

13. HMM ↔ PCFG
How to choose the model parameters that best explain the observed data? ↔ How to choose rule probabilities which maximize the probabilities of the observed sentences? ↔

14. Interesting Probabilities
What is the probability of having a NP at this position such that it will derive “the building” ? -
Inside Probabilities NP
The gunman sprayed the building with bullets
Outside Probabilities
What is the probability of starting from N1 and deriving “The gunman sprayed”, a NP and “with bullets” ? -

15. Interesting Probabilities
Random variables to be considered
The non-terminal being expanded E.g., NP
The word-span covered by the non-terminal.
E.g., (4,5) refers to words “the building”
While calculating probabilities, consider:
The rule to be used for expansion : E.g., NP  DT NN
The probabilities associated with the RHS non-terminals : E.g., DT subtree’s inside/outside probabilities & NN subtree’s inside/outside probabilities

16. Outside Probabilities
Forward ↔ Outside probabilities
j(p,q) : The probability of beginning with N1 & generating the non-terminal Njpq and all words outside wp..wq
Forward probability :
Outside probability : N1  Nj
w1 ………wp-1wp…wqwq+1 ……… wm

17. Inside Probabilities Backward ↔ Inside probabilities
j(p,q) : The probability of generating the words wp..wq starting with the non-terminal Njpq.
Backward probability :
Inside probability : N1  Nj 
w1 ………wp-1wp…wqwq+1 ……… wm

18. Outside & Inside ProbabilitiesNP
The gunman sprayed the building with bullets

19. Inside probabilities j(p,q)
Base case:
Base case is used for rules which derive the words or terminals directly
E.g., Suppose Nj = NN is being considered & NN  building is one of the rules with probability 0.5

20. Induction Step
Induction step : Nj Nr Ns wp wd
wd+1 wq
Consider different splits of the words - indicated by d E.g., the huge building
Consider different non-terminals to be used in the rule: NP  DT NN, NP  DT NNS are available options Consider summation over all these.
Split here for d= d=3

21. The Bottom-Up Approach
The idea of induction
Consider “the gunman”
Base cases : Apply unary rules
DT  the Prob = 1.0
NN  gunman Prob = 0.5
Induction : Prob that a NP covers these 2 words
= P (NP  DT NN) * P (DT deriving the word “the”) * P (NN deriving the word “gunman”)
= 0.5 * 1.0 * 0.5 = 0.25
NP0.5
DT1.0
NN0.5
The gunman

22. Parse Triangle A parse triangle is constructed for calculating j(p,q)
Probability of a sentence using j(p,q):

23. Parse Triangle Fill diagonals with The (1) gunman (2) sprayed (3)
building (5)
with (6)
bullets (7) 1 2 3 4 5 6 7
Fill diagonals with

24. Parse Triangle Calculate using induction formula The (1) gunman (2)
sprayed (3)
the (4)
building (5)
with (6)
bullets (7) 1 2 3 4 5 6 7
Calculate using induction formula

25. Example Parse t1` The gunman sprayed the building with bullets. S1.0
Rule used here is
VP  VP PP
NP0.5
VP0.6
NN0.5
DT1.0
VP0.4
PP1.0
P1.0
NP0.3
NP0.5
The
gunman
VBD1.0
DT1.0
NN0.5
with
NNS1.0
sprayed
the
building
bullets

26. Another Parse t2 The gunman sprayed the building with bullets. S1.0
Rule used here is
VP  VBD NP
NP0.5
VP0.4
NN0.5
DT1.0
VBD1.0
NP0.2
The
gunman
sprayed
NP0.5
PP1.0
DT1.0
NN0.5
P1.0
NP0.3
the
building
with
NNS1.0
bullets

27. Parse Triangle The (1) gunman (2) sprayed (3) the (4) building (5)
with (6)
bullets (7) 1 2 3 4 5 6 7

28. Different Parses Consider
Different splitting points :
E.g., 5th and 3rd position
Using different rules for VP expansion :
E.g., VP  VP PP, VP  VBD NP
Different parses for the VP “sprayed the building with bullets” can be constructed this way.

29. Outside Probabilities j(p,q)
Base case:
Inductive step for calculating : N1
Nfpe
Njpq
Ng(q+1)e
Summation over f, g & e w1
wp-1 wp wq
wq+1 we
we+1 wm

30. Probability of a Sentence
Joint probability of a sentence w1m and that there is a constituent spanning words wp to wq is given as: N1 NP
The gunman sprayed the building with bullets

31. Top Down Bottom Up Parsing
for
Structurally ambiguous sentences

32. Top Down Bottom Up Chart Parsing for Structurally Ambiguous Sentences
Sentence “I saw a boy with a telescope”
Grammar:
S  NP VP
NP  ART N | ART N PP | PRON
VP  V NP PP | V NP
ART  a | an | the
N  boy | telescope
PRON  I
V  saw 32

33. Chart Parsing
I saw a boy with a telescope 33

34. Chart Parsing I saw a boy with a telescope 1 2 3 4 5 6 7 8 34
S1? NP12VP2?
NP12  PRON12
NP13 ART12N23
NP1? ART12N23PP3? 34

35. Chart Parsing I saw a boy with a telescope 1 2 3 4 5 6 7 8 35 NP
S1? NP12VP2?
NP12  PRON12
NP12  PRON12
NP13 ART12N23
NP1? ART12N23PP3?
S1? NP12VP2?
VP2?V23NP3?
VP2?V23NP3?PP?? NP 35

36. Chart Parsing I saw a boy with a telescope 1 2 3 4 5 6 7 8 36 NP
S1? NP12VP2?
VP2?V23NP3?PP??
VP2?V23NP3?
NP12  PRON12
NP12  PRON12
NP13 ART12N23
NP1? ART12N23PP3?
S1? NP12VP2?
VP2?V23NP3?
VP2?V23NP3?PP??
NP3?ART34N45PP5?
NP35ART34N45 NP 36

37. Chart Parsing I saw a boy with a telescope 1 2 3 4 5 6 7 8 37 NP
S1? NP12VP2?
VP2?V23NP3?PP??
VP2?V23NP3?
NP35 ART34N45
NP3?ART34N45PP5?
NP12  PRON12
NP12  PRON12
NP13 ART12N23
NP1? ART12N23PP3?
S1? NP12VP2?
VP2?V23NP3?
VP2?V23NP3?PP??
NP3?ART34N45PP5?
NP35ART34N45 NP 37

38. Chart Parsing I saw a boy with a telescope 1 2 3 4 5 6 7 8 38 NP
S1? NP12VP2?
VP2?V23NP3?PP??
VP2?V23NP3?
NP35 ART34N45
NP3?ART34N45PP5?
NP12  PRON12
NP12  PRON12
NP13 ART12N23
NP1? ART12N23PP3?
S1? NP12VP2?
VP2?V23NP3?
VP2?V23NP3?PP??
NP3?ART34N45PP5?
NP35ART34N45
NP35ART34N45
NP3?ART34N45PP5?
PP5?P56NP6? NP 38

39. Chart Parsing I saw a boy with a telescope 1 2 3 4 5 6 7 8 39 NP NP
S1? NP12VP2?
VP2?V23NP3?PP??
VP2?V23NP3?
NP35 ART34N45
NP3?ART34N45PP5?
NP12  PRON12
NP12  PRON12
NP13 ART12N23
NP1? ART12N23PP3?
S1? NP12VP2?
VP2?V23NP3?
VP2?V23NP3?PP??
NP3?ART34N45PP5?
NP35ART34N45
NP35ART34N45
NP3?ART34N45PP5?
PP5?P56NP6? NP NP 39

40. Chart Parsing I saw a boy with a telescope 1 2 3 4 5 6 7 8 40 NP NP
S1? NP12VP2?
VP2?V23NP3?PP??
VP2?V23NP3?
NP35 ART34N45
NP3?ART34N45PP5?
PP5?P56NP6?
NP12  PRON12
NP12  PRON12
NP13 ART12N23
NP1? ART12N23PP3?
S1? NP12VP2?
VP2?V23NP3?
VP2?V23NP3?PP??
NP3?ART34N45PP5?
NP35ART34N45
NP35ART34N45
NP3?ART34N45PP5?
PP5?P56NP6?
NP67PRON67
NP68 ART67N78
NP6? ART67N78PP8? NP NP 40

41. Chart Parsing I saw a boy with a telescope 1 2 3 4 5 6 7 8 41 NP NP
S1? NP12VP2?
VP2?V23NP3?PP??
VP2?V23NP3?
NP35 ART34N45
NP3?ART34N45PP5?
PP5?P56NP6?
NP68ART67N78
NP6?ART67N78PP8?
NP12  PRON12
NP12  PRON12
NP13 ART12N23
NP1? ART12N23PP3?
S1? NP12VP2?
VP2?V23NP3?
VP2?V23NP3?PP??
NP3?ART34N45PP5?
NP35ART34N45
NP35ART34N45
NP3?ART34N45PP5?
PP5?P56NP6?
NP67PRON67
NP68 ART67N78
NP6? ART67N78PP8? NP NP 41

42. Chart Parsing I saw a boy with a telescope 1 2 3 4 5 6 7 8 42 NP NP NP
S1? NP12VP2?
VP2?V23NP3?PP??
VP2?V23NP3?
NP35 ART34N45
NP3?ART34N45PP5?
PP5?P56NP6?
NP68ART67N78
NP6?ART67N78PP8?
NP68ART67N78
NP12  PRON12
NP12  PRON12
NP13 ART12N23
NP1? ART12N23PP3?
S1? NP12VP2?
VP2?V23NP3?
VP2?V23NP3?PP??
NP3?ART34N45PP5?
NP35ART34N45
NP35ART34N45
NP3?ART34N45PP5?
PP5?P56NP6?
NP67PRON67
NP68 ART67N78
NP6? ART67N78PP8?
PP5?P56NP6? NP NP NP 42

43. Chart Parsing I saw a boy with a telescope 1 2 3 4 5 6 7 8 43 NP NP NP
S1? NP12VP2?
VP2?V23NP3?PP??
VP2?V23NP3?
NP35 ART34N45
NP3?ART34N45PP5?
PP5?P56NP6?
NP68ART67N78
NP6?ART67N78PP8?
NP68ART67N78
NP12  PRON12
NP12  PRON12
NP13 ART12N23
NP1? ART12N23PP3?
S1? NP12VP2?
VP2?V23NP3?
VP2?V23NP3?PP??
NP3?ART34N45PP5?
NP35ART34N45
NP35ART34N45
NP3?ART34N45PP5?
PP5?P56NP6?
NP67PRON67
NP68 ART67N78
NP6? ART67N78PP8?
PP5?P56NP6? NP NP NP PP 43

44. Chart Parsing I saw a boy with a telescope 1 2 3 4 5 6 7 8 44 NP NP NP
S1? NP12VP2?
VP2?V23NP3?PP??
VP2?V23NP3?
NP35 ART34N45
NP3?ART34N45PP5?
PP5?P56NP6?
NP68ART67N78
NP6?ART67N78PP8?
NP68ART67N78
NP12  PRON12
NP12  PRON12
NP13 ART12N23
NP1? ART12N23PP3?
S1? NP12VP2?
VP2?V23NP3?
VP2?V23NP3?PP??
NP3?ART34N45PP5?
NP35ART34N45
NP35ART34N45
NP3?ART34N45PP5?
PP5?P56NP6?
NP67PRON67
NP68 ART67N78
NP6? ART67N78PP8?
PP5?P56NP6? NP NP NP PP NP 44

45. Chart Parsing I saw a boy with a telescope 1 2 3 4 5 6 7 8 45 NP NP NP
S1? NP12VP2?
VP2?V23NP3?PP??
VP2?V23NP3?
NP35 ART34N45
NP3?ART34N45PP5?
PP5?P56NP6?
NP68ART67N78
NP6?ART67N78PP8?
NP68ART67N78
NP12  PRON12
NP12  PRON12
NP13 ART12N23
NP1? ART12N23PP3?
S1? NP12VP2?
VP2?V23NP3?
VP2?V23NP3?PP??
NP3?ART34N45PP5?
NP35ART34N45
NP35ART34N45
NP3?ART34N45PP5?
PP5?P56NP6?
NP67PRON67
NP68 ART67N78
NP6? ART67N78PP8?
PP5?P56NP6? NP NP NP PP NP VP 45

46. Chart Parsing I saw a boy with a telescope 1 2 3 4 5 6 7 8 46 NP NP NP
S1? NP12VP2?
VP2?V23NP3?PP??
VP2?V23NP3?
NP35 ART34N45
NP3?ART34N45PP5?
PP5?P56NP6?
NP68ART67N78
NP6?ART67N78PP8?
NP68ART67N78
NP12  PRON12
NP12  PRON12
NP13 ART12N23
NP1? ART12N23PP3?
S1? NP12VP2?
VP2?V23NP3?
VP2?V23NP3?PP??
NP3?ART34N45PP5?
NP35ART34N45
NP35ART34N45
NP3?ART34N45PP5?
PP5?P56NP6?
NP67PRON67
NP68 ART67N78
NP6? ART67N78PP8?
PP5?P56NP6? NP NP NP PP NP VP S 46