1. Introduction and Jurafsky Model Resource: A Probabilistic Model of Lexical and Syntactic Access and Disambiguation, Jurafsky 1996

2.  Process of accessing words? or  What happens after all the words are recognized?  Analyze it  Evaluate it  Creating new knowledge  Thinking guided by print?  Psycholinguistics

3.  What is constructed when we comprehend a sentence?  Propositional representation  What role do words play?  Mental lexicon  How does the process of constructing representation occur?  How is the ability to construct a representation acquired?

4.  Reading time: total time by a reader to read a peace of text  Reading time experiments  Word by word reading  Sentence by sentence reading  Eye tracking experiment

5. Source: https://wiki.brown.edu/confluence/display/kertzlab/Eye-Tracking+While+Reading Eye Tracking While Reading (ETWR) Experiment

6. Reading time is higher for ambiguous sentences

7.  Probabilistic Context Free Grammar (PCFG) model  Entropy Reduction model  Competition-Integration model  Connectionist model

8.  Probabilistic Context Free Grammar (PCFG) model  Entropy Reduction model  Competition-Integration model  Connectionist model

9.  Assumptions by Jurafsky (1996)  Observed preferences in interpretation of ambiguous sentences reflect probabilities of different syntactic structures.  Processing difficulty is a continuum ▪ Slight preferences at one end ▪ Garden path constructs at another end  Several types of ambiguities  Lexical category ambiguity  Attachment ambiguity  Unexpected thematic fit  Main clause vs. reduced relative clause ambiguity

10.  Garden path: to be led down to the garden path  To be misled or deceived  A garden path sentence [Wikipedia]  Grammatically correct sentence  Starts in such a way that reader’s most likely interpretation will be incorrect  Reader will be lured to a dead-end parse  Garden path sentences will be marked with # now on

11. (1) #The old man the boat. (2) #The horse raced (3) #The complex houses married and single soldiers and their families. (4) #The government plans to raise taxes were defeated. past the barnfell.

12.  Ambiguity resolved without trouble (fires = N or V): 1a. The warehouse fires destroyed all the buildings 1b. The warehouse fires a dozen employees each year.  Ambiguity leads to garden path (complex= N or Adj, houses= N or V, etc.): 2a. #The complex houses married and single students. 2b. #The old man the boats

13. (1) The spy saw the policeman with binocular (2) The spy saw the policeman with a revolver (3) The bird saw the birdwatcher with binocular

14.  Prepositional phrase can attach to NP or VP. 1. I saw the man with the glasses

15.  The arguments required by a verb are its subcategorization frame or valence.  Primary arguments  Secondary arguments  Attachment preferences vary between verbs (1)The women discussed the dogs on the beach. a.The women discussed the dogs which were on the beach. (90%) b.The women discussed them (the dogs) while on the beach. (10%) (2)The women kept the dogs on the beach. a.The women kept the dogs which were on the beach. (5%) b.The women kept them (the dogs) on the beach. (95%)

16. (1) The cop arrested by the detective was guilty of taking bribes (2) The crook arrested by the detective was guilty of taking bribes  (1) introduces more disambiguation difficulty as the initial noun phrase (the cop) is a good agent of the first verb (arrested)

17.  Reduced relative clause: that-clause without the that. a. #The horse raced past the barn fell. a’. #The horse (that) raced past the barn fell. b. The horse found in the woods died b’. The horse (that was) found in the woods died  Another case of different subcategorization preferences:  X raced >> X raced Y (intransitive preferred over transitive)  X found Y >> X found (transitive preferred over intransitive)

18.  if multiple rules can apply, choose one based on a selection rule  determinism  example selection rule: minimal attachment (choose the tree with the fewest nodes).  if parse fails, backtrack to choice point and reparse  backtracking occurs, causes increased processing times.

19.  if multiple rules can apply, pursue all possibilities in parallel  non-determinism  if any parse fails, discard it;  problem: number of parse trees can grow exponentially  solution: only pursue a limited number of possibilities (bounded parallelism).  Prune some of the unpromising parses  garden path means correct tree was pruned from search space;  backtracking occurs, causes increased processing times.

20.  How to model non-determinism  Probabilistic parsing  Parsing model [Jurafsky (1996)]  Each full or partial parse is assigned a probability.  Parses are pruned from the search space if their probability is a factor of α below the most probable parse (beam search).  How are parse probabilities determined?

22.  Grammar for a language is a set of rewrite rules Non-terminal Symbol Terminal Symbol Context-Free Context-Sensitive

29. Simple PCFG S → NP VP S → Aux NP VP S → VP NP → Pronoun NP → Proper-Noun NP → Det Nominal Nominal → Noun Nominal → Nominal Noun Nominal → Nominal PP VP → Verb VP → Verb NP VP → VP PP PP → Prep NP Grammar 0.8 0.1 0.2 0.6 0.3 0.2 0.5 0.2 0.5 0.3 1.0 Prob + + + + 1.0 Det → the | a | that | this 0.6 0.2 0.1 0.1 Noun → book | flight | meal | money 0.1 0.5 0.2 0.2 Verb → book | include | prefer 0.5 0.2 0.3 Pronoun → I | he | she | me 0.5 0.1 0.1 0.3 Proper-Noun → Houston | NWA 0.8 0.2 Aux → does 1.0 Prep → from | to | on | near | through 0.25 0.25 0.1 0.2 0.2 Lexicon

30. Probabilistic Grammar Conversion S → NP VP S → Aux NP VP S → VP NP → Pronoun NP → Proper-Noun NP → Det Nominal Nominal → Noun Nominal → Nominal Noun Nominal → Nominal PP VP → Verb VP → Verb NP VP → VP PP PP → Prep NP Original Grammar Chomsky Normal Form S → NP VP S → X1 VP X1 → Aux NP S → book | include | prefer 0.01 0.004 0.006 S → Verb NP S → VP PP NP → I | he | she | me 0.01 0.02 0.02 0.06 NP → Houston | NWA 0.16.04 NP → Det Nominal Nominal → book | flight | meal | money 0.03 0.15 0.06 0.06 Nominal → Nominal Noun Nominal → Nominal PP VP → book | include | prefer 0.1 0.04 0.06 VP → Verb NP VP → VP PP PP → Prep NP 0.8 0.1 0.2 0.6 0.3 0.2 0.5 0.2 0.5 0.3 1.0 0.8 0.1 1.0 0.05 0.03 0.6 0.2 0.5 0.3 1.0

32.  Subcategorization frames of the verb keep:  Valence probabilities tell us how likely each of these frames is.

33.  Like PCFG probabilities, valence probabilities are estimated from a treebank.

34. Garden path caused by construction probabilities

36. Disambiguation using construction probabilities, no garden path “The warehouse fires destroyed all the buildings”

37. Disambiguation using construction probabilities, no garden path “The warehouse fires a dozen employees each year.”

38. Disambiguation using valence probabilities, no garden path: “keep the dogs on the beach”

39. Disambiguation using valence probabilities, no garden path: “keep the dogs on the beach”

40. Disambiguation using valence probabilities, no garden path: “discuss the dogs on the beach”

41. Disambiguation using valence probabilities, no garden path: “discuss the dogs on the beach”

42. Garden path caused by construction probabilities and valence probabilities: (main verb interpretation) “the horse raced past……”

43. Garden path caused by construction probabilities and valence probabilities: (reduced relative interpretation) “the horse raced past……”

44. Disambiguation using construction probabilities and valence probabilities, no garden path: (main verb) “The bird found in the room died”

45. Disambiguation using construction probabilities and valence probabilities, no garden path: (reduced relative) “The bird found in the room died”

46. Crucial assumption: if the relative probability of a tree falls below a certain value, then it will be pruned. Assumption: a garden path occurs if the probability ratio is higher than 5:1.