1 SIMS 256: Applied Natural Language Processing Marti Hearst November 27, 2006

2 Outline Discourse Processing Going beyond the sentence Characteristics Issues: Segmentation –Linear –Hierarchical Co-reference / anaphora resolution Dialogue Processing

3 Adapted from slide by Julia Hirschberg What makes a text/dialogue coherent? “Consider, for example, the difference between passages (18.71) and (18.72). Almost certainly not. The reason is that these utterances, when juxtaposed, will not exhibit coherence. Do you have a discourse? Assume that you have collected an arbitrary set of well-formed and independently interpretable utterances, for instance, by randomly selecting one sentence from each of the previous chapters of this book.” vs….

4 Adapted from slide by Julia Hirschberg What makes a text/dialogue coherent? “Assume that you have collected an arbitrary set of well-formed and independently interpretable utterances, for instance, by randomly selecting one sentence from each of the previous chapters of this book. Do you have a discourse? Almost certainly not. The reason is that these utterances, when juxtaposed, will not exhibit coherence. Consider, for example, the difference between passages (18.71) and (18.72). (J&M:695)

5 Adapted from slide by Julia Hirschberg What makes a text coherent? Discourse/topic structure Appropriate sequencing of subparts of the discourse Rhetorical structure Appropriate use of coherence relations between subparts of the discourse Referring expressions Words or phrases, the semantic interpretation of which is a discourse entity

6 Adapted from slide by Julia Hirschberg Information Status Contrast –John wanted a poodle but Becky preferred a corgi. Topic/comment –The corgi they bought turned out to have fleas. Theme/rheme –The corgi they bought turned out to have fleas. Focus/presupposition –It was Becky who took him to the vet. Given/new –Some wildcats bite, but this wildcat turned out to be a sweetheart. –Contrast Speaker (S) and Hearer (H)

7 Adapted from slide by Julia Hirschberg Entities when first introduced are new Brand-new (H must create a new entity) I saw a dinosaur today. Unused (H already knows of this entity) I saw your mother today. Evoked entities are old -- already in the discourse Textually evoked The dinosaur was scaley and gray. Situationally evoked The light was red when you went through it. Inferrables Containing I bought a carton of eggs. One of them was broken. Non-containing A bus pulled up beside me. The driver was a monkey. Determining Given vs. New

8 Adapted from slide by Julia Hirschberg Given/New and Definiteness/Indefiniteness Subject NPs tend to be syntactically definite and old Object NPs tend to be indefinite and new I saw a black cat yesterday. The cat looked hungry. –Definite articles, demonstratives, possessives, personal pronouns, proper nouns, quantifiers like all, every Indefinite articles, quantifiers like some, any, one signal indefiniteness…but…. This guy came into the room

9 Discourse/Topic Structure Text Segmentation: Linear –TextTiling –Look for changes in content words Hierarchical –Grosz & Sidner’s Centering theory –Morris & Hirst’s algorithm –Lexical chaining through Roget’s thesaurus Hierarchical + Relations –Mann et al.’s Rhetorical Structure Theory –Marcu’s algorithm

10 TextTiling (Hearst 94) Goal: find multi-paragraph topics Example: 21 paragraph article called Stargazers

11 TextTiling (Hearst 94) Goal: find multi-paragraph topics But … it’s difficult to define topic (Brown & Yule) Focus instead on topic shift or change Change in content, by contrast with setting, scene, characters Mechanism: compare adjacent blocks of text look for shifts in vocabulary

12 Intuition behind TextTiling

13 TextTiling Algorithm Tokenization Lexical Score Determination Blocks Vocabulary Introductions Chains Boundary Identification

14 Tokenization Convert text stream into terms (words) Remove “stop words” Reduce to root (inflectional morphology) Subdivide into “token-sequences” (substitute for sentences) Find potential boundary points (paragraphs breaks)

15 Determining Scores Compute a score at each token-sequence gap Score based on lexical occurrences Block algorithm:

16

17 Boundary Identification Smooth the plot (average smoothing) Assign depth score at each token-sequence gap “Deeper” valleys score higher Order boundaries by depth score Choose boundary cut off (avg-sd/2)

18 Evaluation l Data n Twelve news articles from Dialog n Seven human judges per article n “major” boundaries: chosen by >= 3 judges n Avg number of paragraphs: n Avg number of boundaries: 10 (39%) l Results n Between upper and lower bounds n Upper bound: judges’ averages n Lower bound: reasonable simple algorithm

19 Assessing Agreement Among Judges KAPPA Coefficient n Measures pairwise agreement n Takes expected chance agreement into account n P(A) = proportion of times judges agree n P(E) = proportion expected chance agreement n.43 to.68 (Isard & Carletta 95, boundaries) n.65 to.90 (Rose 95, sentence segmentation) n Here, k=.647

20 TextTiling Conclusions First computational investigation into multi-paragraph discourse units Simple Discourse Cue: position-sensitive term repetition Acceptable performance for some tasks Has been reproduced/used by many researchers Multi-lingual (applied by others to French, German, Arabic)

21 Adapted from slide by Julia Hirschberg What Can Hierarchical Structure Tell Us? Welcome to word processing. That’s using a computer to type letters and reports. Make a typo? No problem. Just back up, type over the mistake, and it’s gone.  And, it eliminates retyping.  And, it eliminates retyping.

22 Adapted from slide by Julia Hirschberg Centering Theory of Discourse Structure (Grosz & Sidner ‘86) A prominent theory of discourse structure Provides for multiple levels of analysis: S’s purpose as well as content of utterances and S and H’s attentional state Identifies only a few, general relations that hold among intentions Often leads to a hierarchical structure Three components: Linguistic structure Intentional structure Attentional structure

23 Example of Hierarchical Analysis (Morris and Hirst ’91)

24

25 Adapted from slide by Julia Hirschberg Rhetorical Structure Theory (Mann, Matthiessen, and Thompson ‘89) One theory of discourse structure, based on identifying relations between parts of the text Identify meaningful units and the relations between them –Clauses and clause-like units that are unequivocally the nucleus or satellite of a rhetorical relation.  Only the midday sun at tropical latitudes is warm enough] [to thaw ice on occasions,] [but any liquid water formed in this way would evaporate almost instantly] [because of the low atmospheric pressure.] Nucleus/satellite notion encodes asymmetry

26 Adapted from slide by Julia Hirschberg Rhetorical Structure Theory Some rhetorical relations: Elaboration (set/member,class/instance/whole/part…) Contrast: multinuclear Condition: Sat presents precondition for N Purpose: Sat presents goal of the activity in N Sequence: multinuclear Result: N results from something presented in Sat Evidence: Sat provides evidence for something claimed in N

27 Adapted from slide by Daniel Marcu Determining high-level relations [Smart cards are not a new phenomenon. 1 ] [They have been in development since the late 1970s and have found major applications in Europe, with more than a quarter of a billion cards made so far. 2 ] [The vast majority of chips have gone into prepaid, disposable telephone cards, but even so the experience gained has reduced manufacturing costs, improved reliability and proved the viability of smart cards. 3 ] [International and national standards for smart cards are well under development to ensure that cards, readers and the software for the many different applications that may reside on them can work together seamlessly and securely. 4 ] [Standards set by the International Organization for Standardization (ISO), for example, govern the placement of contacts on the face of a smart card so that any card and reader will be able to connect. 5 ]

28 Adapted from slide by Daniel Marcu Representing implicit relations [Smart cards are becoming more attractive 2 ] [as the price of microcomputing power and storage continues to drop. 3 ] [They have two main advantages over magnetic- stripe cards. 4 ] [First, they can carry 10 or even 100 times as much information 5 ] [- and hold it much more robustly. 6 ] [Second, they can execute complex tasks in conjunction with a terminal. 7 ]

29 Adapted from slide by Julia Hirschberg What’s the Rhetorical Structure? 1.System: Hello. How may I help you? 2.User: I would like to find out why I was charged for a call? 3.System: What call would you like to inquire about? 4.User: My bill says I made a call to Syncamaloo, Texas, but I’ve never even heard of this town. 5.System: May I have the date of the call that appears on your bill?

30 Adapted from slide by Daniel Marcu Issues for RST Many variations in expression [I have not read this book.] [It was written by Bertrand Russell.] [I have not read this book,] [which was written by Bertrand Russell.] [I have not read this book written by Bertrand Russell.] [I have not read this Bertrand Russell book.] Rhetorical relations are ambiguous [He caught a bad fever] [while he was in Africa.] –Circumstance > Temporal-Same-Time [With its distant orbit, Mars experiences frigid weather conditions.] [Surface temperatures typically average about –60 degrees Celsius at the equator and can dip to –123 degrees C near the poles. ] –Evidence > Elaboration

31 Adapted from slide by Julia Hirschberg Identifying RS Automatically (Marcu ’99) Train a parser on a discourse treebank 90 RS trees, hand-annotated for rhetorical relations Elementary discourse units (edu’s) linked by RR Parser learns to identify N and S and their RR Features: Wordnet-based similarity, lexical, structural Uses discourse segmenter to identify discourse units Trained to segment on hand-labeled corpus (C4.5) Features: 5-word POS window, presence of discourse markers, punctuation, seen a verb?,… Eval: 96-8% accuracy

32 Adapted from slide by Julia Hirschberg Evaluation of parser: Id edu’s: Recall 75%, Precision 97% Id hierarchical structure (2 edu’s related): Recall 71%, Precision 84% Id nucleus/satellite labels: Recall 58%, Precision 69% Id RR: Recall 38%, Precision 45% Later errors due mostly to edu mis-identification Id of hierarchical structure and n/s status comparable to human when hand-labeled edu’s used Hierarchical structure is easier to id than RR Identifying RS Automatically (Marcu ’99)

33 Adapted from slide by Julia Hirschberg Some Problems with RST (cf. Moore & Pollack ‘92) How many Rhetorical Relations are there? How can we use RST in dialogue as well as monologue? RST does not allow for multiple relations holding between parts of a discourse RST does not model overall structure of the discourse

34 Adapted from slide by Ani Nenkova Referring Expressions Referring expressions are words or phrases, the semantic interpretation of which is a discourse entity (also called referent) Discourse entities are semantic objects. Can have multiple syntactic realizations within a text Discourse entities exist in the domain D, in which a text is interpreted

35 Adapted from slide by Ani Nenkova Referring Expressions: Example A pretty woman entered the restaurant. She sat at the table next to mine and only then I recognized her. This was Amy Garcia, my next door neighbor from 10 years ago. The woman has totally changed! Amy was at the time shy…

36 Adapted from slide by Ani Nenkova Pronouns vs. Full NP A pretty woman entered the restaurant. She sat at the table next to mine and only then I recognized her. This was Amy Garcia, my next door neighbor from 10 years ago. The woman has totally changed! Amy was at the time shy…

37 Adapted from slide by Ani Nenkova Definite vs. Indefinite NPs A pretty woman entered the restaurant. She sat at the table next to mine and only then I recognized her. This was Amy Garcia, my next door neighbor from 10 years ago. The woman has totally changed! Amy was at the time shy…

38 Adapted from slide by Ani Nenkova Common Noun vs. Proper Noun A pretty woman entered the restaurant. She sat at the table next to mine and only then I recognized her. This was Amy Garcia, my next door neighbor from 10 years ago. The woman has totally changed! Amy was at the time shy…

39 Adapted from slide by Ani Nenkova Modified vs. Bare head NP A pretty woman entered the restaurant. She sat at the table next to mine and only then I recognized her. This was Amy Garcia, my next door neighbor from 10 years ago. The woman has totally changed! Amy was at the time shy…

40 Adapted from slide by Ani Nenkova Premodified vs. Postmodified A pretty woman entered the restaurant. She sat at the table next to mine and only then I recognized her. This was Amy Garcia, my next door neighbor from 10 years ago. The woman has totally changed! Amy was at the time shy…

41 Adapted from slide by Ani Nenkova Anaphora resolution Finding in a text all the referring expressions that have one and the same denotation Pronominal anaphora resolution Anaphora resolution between named entities Full noun phrase anaphora resolution

42 Adapted from slide by Ani Nenkova Anaphora Resolution A pretty woman entered the restaurant. She sat at the table next to mine and only then I recognized her. This was Amy Garcia, my next door neighbor from 10 years ago. The woman has totally changed! Amy was at the time shy…

43 Adapted from slide by Ani Nenkova Pronominal anaphora resolution Rule-based vs statistical (Ken 1996), (Lap 1994) vs (Ge 1998) Performed on full syntactic parse vs on shallow syntactic parse (Lap 1994), (Ge 1998) vs (Ken 1996) Type of text used for the evaluation (Lap 1994) computer manual texts (86% accuracy) (Ge 1998) WSJ articles (83% accuracy) (Ken 1996) different genres (75% accuracy)

44 Adapted from slide by Ani Nenkova Pronominal anaphora resolution Generic vs specific reference 1. The Vice-President of the United States is also President of the Senate. 2. Historically, he is the President’s key person in negotiations with Congress 3a. He is required to be 35 years old. 3b. As Ambassador to China, he handled many tricky negotiations, so he is well prepared for the job

45 Talking to a Machine….and (often) Getting an Answer Today’s spoken dialogue systems make it possible to accomplish real tasks without talking to a person Key advances Stick to goal-directed interactions in a limited domain Prime users to adopt the vocabulary you can recognize Partition the interaction into manageable stages Judicious use of system vs. mixed initiative

46 Adapted from slide by Julia Hirschberg Acoustic and Prosodic Cues to Discourse Structure Intuition: Speakers vary acoustic and prosodic cues to convey variation in discourse structure Systematic? In read or spontaneous speech? Evidence: Observations from recorded corpora Laboratory experiments Machine learning of discourse structure from acoustic/prosodic features

47 Adapted from slide by Julia Hirschberg Boston Directions Corpus (Hirschberg & Nakatani ’96) Experimental Design –12 speakers: 4 used –Spontaneous and read versions of 9 direction-giving tasks Corpus: 50m read; 67m spon Labeling Prosodic: ToBI intonational labeling Discourse: Grosz & Sidner

48 Adapted from slide by Julia Hirschberg ds1: step 1, enter and get token first enter the Harvard Square T stop and buy a token ds2: inbound on red line then proceed to get on the inbound um Red Line uh subway Boston Directions Corpus: Describe how to get to MIT from Harvard

49 Adapted from slide by Julia Hirschberg ds3: take subway from hs, to cs to ks and take the subway from Harvard Square to Central Square and then to Kendall Square –ds4: describe ks station you’ll see a music sculpture there which will tell you it’s Kendall Square it’s very nice ds5: get off T. then get off the T

50 Dialogue vs. Monologue Monologue and dialogue both involve interpreting Information status Coherence issues Reference resolution Speech acts, implicature, intentionality Dialogue involves managing Turn-taking Grounding and repairing misunderstandings Initiative and confirmation strategies

51 Segmenting Speech into Utterances What is an `utterance’? Why is EOU detection harder than EOS? How does speech differ from text? Single syntactic sentence may span several turns A: We've got you on USAir flight 99 B: Yep A: leaving on December 1. Multiple syntactic sentences may occur in single turn A: We've got you on USAir flight 99 leaving on December. Do you need a rental car? Intonational definitions: intonational phrase, breath group, intonation unit

52 Turns and Utterances Dialogue is characterized by turn-taking: who should talk next, and when they should talk How do we identify turns in recorded speech? Little speaker overlap (around 5% in English -- although depends on domain) But little silence between turns either How do we know when a speaker is giving up or taking a turn? Holding the floor? How do we know when a speaker is interruptable?

53 Simplified Turn-Taking Rule (Sacks et al) At each transition-relevance place (TRP) of each turn: If current speaker has selected A as next speaker, then A must speak next If current speaker does not select next speaker, any other speaker may take next turn If no one else takes next turn, the current speaker may take next turn TRPs are where the structure of the language allows speaker shifts to occur

54 Adjacency pairs set up next speaker expectations GREETING/GREETING QUESTION/ANSWER COMPLIMENT/DOWNPLAYER REQUEST/GRANT ‘Significant silence’ is dispreferred A: Is there something bothering you or not? (1.0s) A: Yes or no? (1.5s) A: Eh? B: No.

55 Turntaking and Initiative Strategies System Initiative S: Please give me your arrival city name. U: Baltimore. S: Please give me your departure city name…. User Initiative S: How may I help you? U: I want to go from Boston to Baltimore on November 8. `Mixed’ initiative S: How may I help you? U: I want to go to Boston. S: What day do you want to go to Boston?

56 Grounding (Clark & Shaefer ‘89) Conversational participants don’t just take turns speaking….they try to establish common ground (or mutual belief) H must ground a S's utterances by making it clear whether or not understanding has occurred How do hearers do this? Several different mechanisms

57 S: I can upgrade you to an SUV at that rate. Continued attention (U gazes appreciatively at S) Relevant next contribution U: Do you have a RAV4 available? Acknowledgement/backchannel U: Ok/Mhmmm/Great! Demonstration/paraphrase U: An SUV. Display/repetition U: You can upgrade me to an SUV at the same rate? Request for repair U: I beg your pardon? Grounding Mechanisms (Clark & Shaefer ‘89)

58 How do we evaluate Dialogue Systems? PARADISE framework (Walker et al ’00) “Performance” of a dialogue system is affected both by what gets accomplished by the user and the dialogue agent and how it gets accomplished Efficiency of the Interaction:User Turns, System Turns, Elapsed Time Quality of the Interaction: ASR rejections, Time Out Prompts, Help Requests, Barge-Ins, Mean Recognition Score (concept accuracy), Cancellation Requests User Satisfaction Task Success: perceived completion, information extracted

59 Identifying Misrecognitions and User Corrections Automatically (Hirschberg, Litman & Swerts) Collect corpus from interactive voice response system Identify speaker ‘turns’ –incorrectly recognized –where speakers first aware of error –that correct misrecognitions Identify prosodic features of turns in each category and compare to other turns Use Machine Learning techniques to train a classifier to make these distinctions automatically

60 Turn Types TOOT: Hi. This is AT&T Amtrak Schedule System. This is TOOT. How may I help you? User: Hello. I would like trains from Philadelphia to New York leaving on Sunday at ten thirty in the evening. TOOT: Which city do you want to go to? User: New York. misrecognition correction aware site

61 Results Reduced error in predicting misrecognized turns to 8.64% Error in predicting ‘awares’ (12%) Error in predicting corrections (18-21%)

62 Adapted from slide by Julia Hirschberg Dialogue Conclusions Spoken dialogue systems presents new problems -- but also new possibilities Recognizing speech introduces a new source of errors Additional information provided in the speech stream offers new information about users’ intended meanings, emotional state (grounding of information, speech acts, reaction to system errors) Why spoken dialogue systems rather than web-based interfaces?