Introduction and Jurafsky Model Resource: A Probabilistic Model of Lexical and Syntactic Access and Disambiguation, Jurafsky 1996

 Process of accessing words? or  What happens after all the words are recognized?  Analyze it  Evaluate it  Creating new knowledge  Thinking guided by print?  Psycholinguistics

 What is constructed when we comprehend a sentence?  Propositional representation  What role do words play?  Mental lexicon  How does the process of constructing representation occur?  How is the ability to construct a representation acquired?

 Reading time: total time by a reader to read a peace of text  Reading time experiments  Word by word reading  Sentence by sentence reading  Eye tracking experiment

Source: Eye Tracking While Reading (ETWR) Experiment

Reading time is higher for ambiguous sentences

 Probabilistic Context Free Grammar (PCFG) model  Entropy Reduction model  Competition-Integration model  Connectionist model

 Probabilistic Context Free Grammar (PCFG) model  Entropy Reduction model  Competition-Integration model  Connectionist model

 Assumptions by Jurafsky (1996)  Observed preferences in interpretation of ambiguous sentences reflect probabilities of different syntactic structures.  Processing difficulty is a continuum ▪ Slight preferences at one end ▪ Garden path constructs at another end  Several types of ambiguities  Lexical category ambiguity  Attachment ambiguity  Unexpected thematic fit  Main clause vs. reduced relative clause ambiguity

 Garden path: to be led down to the garden path  To be misled or deceived  A garden path sentence [Wikipedia]  Grammatically correct sentence  Starts in such a way that reader’s most likely interpretation will be incorrect  Reader will be lured to a dead-end parse  Garden path sentences will be marked with # now on

(1) #The old man the boat. (2) #The horse raced (3) #The complex houses married and single soldiers and their families. (4) #The government plans to raise taxes were defeated. past the barnfell.

 Ambiguity resolved without trouble (fires = N or V): 1a. The warehouse fires destroyed all the buildings 1b. The warehouse fires a dozen employees each year.  Ambiguity leads to garden path (complex= N or Adj, houses= N or V, etc.): 2a. #The complex houses married and single students. 2b. #The old man the boats

(1) The spy saw the policeman with binocular (2) The spy saw the policeman with a revolver (3) The bird saw the birdwatcher with binocular

 Prepositional phrase can attach to NP or VP. 1. I saw the man with the glasses

 The arguments required by a verb are its subcategorization frame or valence.  Primary arguments  Secondary arguments  Attachment preferences vary between verbs (1)The women discussed the dogs on the beach. a.The women discussed the dogs which were on the beach. (90%) b.The women discussed them (the dogs) while on the beach. (10%) (2)The women kept the dogs on the beach. a.The women kept the dogs which were on the beach. (5%) b.The women kept them (the dogs) on the beach. (95%)

(1) The cop arrested by the detective was guilty of taking bribes (2) The crook arrested by the detective was guilty of taking bribes  (1) introduces more disambiguation difficulty as the initial noun phrase (the cop) is a good agent of the first verb (arrested)

 Reduced relative clause: that-clause without the that. a. #The horse raced past the barn fell. a’. #The horse (that) raced past the barn fell. b. The horse found in the woods died b’. The horse (that was) found in the woods died  Another case of different subcategorization preferences:  X raced >> X raced Y (intransitive preferred over transitive)  X found Y >> X found (transitive preferred over intransitive)

 if multiple rules can apply, choose one based on a selection rule  determinism  example selection rule: minimal attachment (choose the tree with the fewest nodes).  if parse fails, backtrack to choice point and reparse  backtracking occurs, causes increased processing times.

 if multiple rules can apply, pursue all possibilities in parallel  non-determinism  if any parse fails, discard it;  problem: number of parse trees can grow exponentially  solution: only pursue a limited number of possibilities (bounded parallelism).  Prune some of the unpromising parses  garden path means correct tree was pruned from search space;  backtracking occurs, causes increased processing times.

 How to model non-determinism  Probabilistic parsing  Parsing model [Jurafsky (1996)]  Each full or partial parse is assigned a probability.  Parses are pruned from the search space if their probability is a factor of α below the most probable parse (beam search).  How are parse probabilities determined?

 Grammar for a language is a set of rewrite rules Non-terminal Symbol Terminal Symbol Context-Free Context-Sensitive

Simple PCFG S → NP VP S → Aux NP VP S → VP NP → Pronoun NP → Proper-Noun NP → Det Nominal Nominal → Noun Nominal → Nominal Noun Nominal → Nominal PP VP → Verb VP → Verb NP VP → VP PP PP → Prep NP Grammar Prob Det → the | a | that | this Noun → book | flight | meal | money Verb → book | include | prefer Pronoun → I | he | she | me Proper-Noun → Houston | NWA Aux → does 1.0 Prep → from | to | on | near | through Lexicon

Probabilistic Grammar Conversion S → NP VP S → Aux NP VP S → VP NP → Pronoun NP → Proper-Noun NP → Det Nominal Nominal → Noun Nominal → Nominal Noun Nominal → Nominal PP VP → Verb VP → Verb NP VP → VP PP PP → Prep NP Original Grammar Chomsky Normal Form S → NP VP S → X1 VP X1 → Aux NP S → book | include | prefer S → Verb NP S → VP PP NP → I | he | she | me NP → Houston | NWA NP → Det Nominal Nominal → book | flight | meal | money Nominal → Nominal Noun Nominal → Nominal PP VP → book | include | prefer VP → Verb NP VP → VP PP PP → Prep NP

 Subcategorization frames of the verb keep:  Valence probabilities tell us how likely each of these frames is.

 Like PCFG probabilities, valence probabilities are estimated from a treebank.

Garden path caused by construction probabilities

Disambiguation using construction probabilities, no garden path “The warehouse fires destroyed all the buildings”

Disambiguation using construction probabilities, no garden path “The warehouse fires a dozen employees each year.”

Disambiguation using valence probabilities, no garden path: “keep the dogs on the beach”

Disambiguation using valence probabilities, no garden path: “keep the dogs on the beach”

Disambiguation using valence probabilities, no garden path: “discuss the dogs on the beach”

Disambiguation using valence probabilities, no garden path: “discuss the dogs on the beach”

Garden path caused by construction probabilities and valence probabilities: (main verb interpretation) “the horse raced past……”

Garden path caused by construction probabilities and valence probabilities: (reduced relative interpretation) “the horse raced past……”

Disambiguation using construction probabilities and valence probabilities, no garden path: (main verb) “The bird found in the room died”

Disambiguation using construction probabilities and valence probabilities, no garden path: (reduced relative) “The bird found in the room died”

Crucial assumption: if the relative probability of a tree falls below a certain value, then it will be pruned. Assumption: a garden path occurs if the probability ratio is higher than 5:1.