Statistical NLP Winter 2009 Lecture 12: Computational Psycholinguistics Roger Levy

NLP techniques, human parsing Our “parsing” here is about Treebank parsing Now for a bit about human parsing! Techniques from NLP are still the foundation We’ll focus on rational models of human sentence processing [rational = using all available information to make inferences] incremental inference: understanding of and response to a partial utterance

Incrementality and Rationality Online sentence comprehension is hard But lots of information sources can be usefully brought to bear to help with the task Therefore, it would be rational for people to use all the information available, whenever possible This is what incrementality is We have lots of evidence that people do this often “Put the apple on the towel in the box.” (Tanenhaus et al., 1995)

Anatomy of ye olde garden path sentence The horse raced past the barn fell. It’s weird People fail to understand it most of the time People are more likely to misunderstand it than to understand it properly “What’s a barn fell?” The horse that raced past the barn fell The horse raced past the barn and fell Today I’m going to talk about three outstanding puzzles involving garden-path sentences

Garden paths: What we do understand We have decent models of how this sentence is not understood Incremental probabilistic parsing with beam search (Jurafsky, 1996) Surprisal (Hale, 2001; Levy, 2008): the disambiguating word fell is extremely low probability  alarm signal signals “this doesn’t make sense” to the parser These models are based on rational use of evidential information (data-driven probabilistic inference) Also compatible with gradations in garden-path difficulty (Garnsey et al., 1997; McRae et al., 1998)

Hale, 2001; Levy, 2008; Smith & Levy, 2008: surprisal Let the difficulty of a word be its surprisal given its context: Captures the expectation intuition: the more we expect an event, the easier it is to process Many probabilistic formalisms, including probabilistic context-free grammars, can give us word surprisals

a man arrived yesterday 0.3 S  S CC S 0.15 VP  VBD ADVP 0.7 S  NP VP 0.4 ADVP  RB 0.35 NP  DT NN Total probability: 0.7*0.35*0.15*0.3*0.03*0.02*0.4*0.07= 1.85  Algorithms by Lafferty and Jelinek (1992), Stolcke (1995) give us P(w i |context) from a PCFG

Surprisal and garden paths: theory Revisiting the horse raced past the barn fell After the horse raced past the barn, assume 2 parses: Jurafsky 1996 estimated the probability ratio of these parses as 82:1 The surprisal differential of fell in reduced versus unreduced conditions should thus be log 2 83 = 6.4 bits *(assuming independence between RC reduction and main verb)

Surprisal and garden paths: practice An unlexicalized PCFG (from Brown corpus) gets right monotonicity of surprisals at disambiguating word “fell” this is the key comparison; the difference is small, but in the right direction Aside: These are way too high, but that’s because the grammar’s crude

Garden Paths: What we don’t understand so well How do people arrive at the misinterpretations they come up with? What factors induce them to be more or less likely to come up with such a misinterpretation

Outstanding puzzle: length effects Try to read this: Tom heard the gossip about the neighbors wasn’t true. Compare it with this: Tom heard the gossip wasn’t true. Likewise: While the man hunted the deer that was brown and graceful ran into the woods. While the man hunted the deer ran into the woods. The longer the ambiguous region, the harder it is to recover (Frazier & Rayner, 1987; Tabor & Hutchins, 2004) Also problematic for rational models: effects of irrelevant information

Memory constraints in human parsing Sentence meaning is structured The number of logically possible analyses for a sentence is at best exponential in sentence length So we must be entertaining some limited subset of analyses at all times* *“Dynamic programming”, you say? Ask later.

Dynamic programming Exact probabilistic inference with context-free grammars can be done efficiently in O(n 3 ) But… This inference requires strict probabilistic locality Human parsing is linear—that is, O(n)—anyway Here, we’ll explore an approach from the machine- learning literature: the particle filter

The particle filter: general picture Sequential Monte Carlo for incremental observations Let x i be hidden data, z i be unobserved states For parsing: x i are words, z i are structural analyses Suppose that after n-1 observations we have the distribution overinterpretations P(z n-1 |x 1…n-1 ) After obtaining the next word x n, represent the next distribution P(z n |x 1…n ) inductively: Representing P(z i |x 1…i ) by samples makes it a Monte Carlo method

Particle filter with probabilistic grammars S  NP VP1.0V  broke0.3 NP  N0.8V  tired0.3 NP  N RRC0.2Part  raced0.1 RRC  Part Adv1.0Part  broken0.5 VP  V Adv1.0Part  tired0.4 N  horses1.0Adv  quickly1.0 V  raced 0.4 S horses raced quickly VP N VAdv * NP * * * * * * * * * horsesracedquickly RRC N VAdv * * * * * * * * * * * tired * * VP V S * * * 1.0 NP

Returning to the puzzle A-STom heard the gossip wasn’t true. A-LTom heard the gossip about the neighbors wasn’t true. U-STom heard that the gossip wasn’t true. U-LTom heard that the gossip about the neighbors wasn’t true. Previous empirical finding: ambiguity induces difficulty… …but so does the length of the ambiguous region Our linking hypothesis: The proportion of parse failures at the disambiguating region should be monotonically related to the difficulty of the sentence Frazier & Rayner,1982; Tabor & Hutchins, 2004

Model Results Ambiguity matters… But the length of the ambiguous region also matters!

Human results (offline rating study)

Rational comprehension’s other successes Global disambiguation preferences (Jurafsky, 1996) The women discussed the dogs on the beach Basic garden-path sentences (Hale, 2001) The horse raced past the barn fell Garden-path gradience (Narayanan & Jurafsky, 2002) The crook arrested by the detective was guilty Predictability in unambiguous contexts (Levy, 2008) The children went outside to… Grounding in optimality/rational analysis (Norris, 2006; Smith & Levy, 2008) ? ? (that was)(that was) (not difficult)(not difficult) play chat

Behavioral correlates (Tabor et al., 2004) Also, Konieczny (2006, 2007) found compatible results in stops-making- sense and visual-world paradigms These results are problematic for theories requiring global contextual consistency (Frazier, 1987; Gibson, 1991, 1998; Jurafsky, 1996; Hale, 2001, 2006) harder than thrown tossed