1. Speech-to-Speech Translation with Clarifications Julia Hirschberg, Svetlana Stoyanchev Columbia University September 18, 2013

2. Outline Main Problem Key Ideas Solution Details Impact Issues, Gaps, and Future work

3. Speech Translation Speech-to-Speech translation system 3 L1 Speaker lation Speech Question (L1) Translated Question (L2) Answer (L2) Translated Answer (L1) L2 Speaker Translation System

4. Speech Translation Translation may be impaired by: Speech recognition errors Word Error rate in English side of Transtac is 9% Word error rate in Let’s Go bus information is 50% A speaker may use ambiguous language A speech recognition error may be caused by use of out-of-vocabulary words 4

5. Translation System Speech Translation Speech-to-Speech translation system Introduce a clarification component 5 L1 Speaker Speech Question (L1) Translated Question (L2) Answer (L2) Translated Answer (L1)) Clarification sub-dialogue Clarification sub-dialogue L2 Speaker Dialogue Manager

6. Key Ideas Use targeted clarifications Address challenges with targeted clarifications Data collection for system evaluation

7. Most Common Clarification Strategies in Dialogue Systems “Please repeat” “Please rephrase” System repeats the previous question 7

8. What Clarification Questions Do Human Speakers Ask? Targeted reprise questions (M. Purver) o Ask a targeted question about the part of an utterance that was misheard or misunderstood, including understood portions of the utterance o Speaker: Do you have anything other than these XXX plans? o Non-Reprise: What did you say?/Please repeat. o Reprise: What kind of plans? 88% of human clarification questions are reprise 12% non-reprise Goal: Introduce targeted (reprise) questions into a spoken system 8

9. Advantages of Targeted Clarifications More natural User does not have to repeat the whole utterance/command Provides grounding and implicit confirmation  Speech-to-speech translation  Useful in systems that handle natural language user responses/commands/queries and a wide range of topics and vocabulary  Tutoring system  Virtual assistants (in car, in home): a user command may contain ASR error due to noise, background speech, etc. 9

10. Types of Clarification Questions in the TBOLT System Rephrase part Used when an error is OOV and NOT a name (works on difficult non-OOV words as well) Asks to rephrase the error segment “I did not understand when you said: fiscal. Please give me another word or phrase for it.” Spelling Used for names “Please spell ‘Rockefeller’.” Disambiguation Used to disambiguate between homophones “Did you mean plain as in extensive tract of level open land, or, plane as in an aircraft ?”

11. Types of Questions (cont.) Reprise (as found in human-human communication) o Repeats part of the utterance before the error segment o User: We will search some of the XXX to make sure everyone is safe. o System: We will search some of the what? Reprise/ Rephrase-part o Combines a targeted question with a rephrase question o System: We will search some of the what? Please say another word or phrase for this: ‘vehicles’. Confirmation o A yes/no question to confirm an utterance o “Did you say ‘the breach is located here’?”

12. Requirement for a Targeted Question Error Detection  Error segment boundaries  Error type Does the error contain a proper name? Does the error contain an out-of-vocabulary (OOV) word?

13. Challenges with Using Targeted Clarifications 1)A system's clarification question may not be appropriate because  An error segment may not be detected correctly  An error type is not identified correctly 2)How should a system construct a targeted/reprise clarification question 3)How long should the system persist in its clarification? Our goal is to address these challenges empirically 13

14. Examples of Inappropriate Clarification Questions Rephrase-part (incorrect error type detected) User: Captain Bell wants to ask you some questions. Error Detection: Captain to ask you some questions. System: I did not understand when you said "Bell wants". Please give me another word or phrase for it. Reprise/ Targeted Rephrase (error segment is identified incorrectly) User: The insurgents are denouncing us. ASR/Error detection: The insurgents are doing now and System: The insurgents are doing now and what? Please say another word or phrase for "cing us".

15. Examples (cont.) Spelling (error type is incorrectly detected as a name) User: It sounds like there is a kernel of truth to your story. Error Detection: It sounds like there is a of truth to your story. System: I think I heard a name when you said this: "kernel". Please spell that name.

16. Examples (cont.) Disambiguation (disambiguated word is misunderstood) User: How often do you have problems with generators? ASR/Error Detection: How often do you have problems with general? System: Did you mean general as in broad or general as in a military officer?

17. Causes of Inappropriate Questions Rephrase part  Partial word is detected as an error  Detected segment contains a name  Detected segment is a function word (to, from, the …) Disambiguate  Neither choice for disambiguation is correct  Disambiguated word is misrecognized Spell  Non-name  Long segment Reprise Contains undetected recognition error

18. Goal Develop a method to automatically identify when an inappropriate question is asked  Use user’s answers to detect if a question was inappropriate

19. Data Collection Simulation clarification system Users were asked to read a sentence and then were played a pre-recorded question Led to believe they were interacting with the actual system

20. Data Collection(cont.) Prepared 228 questions  84 appropriate  144 inappropriate  For each type of clarification questions, create appropriate and inappropriate questions,  Total 19 categories of clarification questions Each subject was asked 144 questions Recorded their initial utterances and their answers to the questions

21. User Responses Subjects tended to be cooperative Answers varied from subject to subject Example: “I did not understand when you said: ‘Betirma’. Please give me another word or phrase for it.” o “No" o "Betirma" o “Betirma bravo echo tango india romeo mike alpha"

22. User Responses (cont.) Example 2:  User: “How often do you have problems with generators?”  System: “Did you mean general as in broad or general as in a military officer?” o "generator as in a machine for making electricity" o "no" o "generators"

23. Method Extract lexical and prosodic features from responses  Number of pauses, speech energy, speech tempo  Lexical and prosodic difference between initial response and an answer to clarification  Measure number of times subjects replay each question  Measure latency: length of pause before answer Determine whether questions are appropriate or inappropriate based on user responses

24. Challenge 2: Constructing Targeted Clarification Questions Previous work: collected clarification questions using mturk (Stoyanchev et al. 2012, 2013) Using human-generated questions manually created a set of generation rules Evaluated generated questions with human subjects

25. Types of Questions R_GEN Generic: what? Applies if no other rules apply Sentence: The doctor will most likely prescribe XXX Question: the doctor will most likely prescribe WHAT? R_SYN Syntactic: about what about ? Applies when: there is VB after error; VB and error share a parent Sentence: When was the XXX contacted? Question: When was WHAT contacted? R_NMOD: which ? Applies when: DEP TAG error = NMOD and parent POS = NN | NNS Sentence: Do you have anything other than these XXX plans Question: Which plans? R_START: what about

26. Evaluation Questionnaire Generated questions automatically using the rules for a set of 84 sentences Asked humans (mturk) to create a clarification questions for the same sentences Questionnaire applied to both human and computer- generated questions

27. Subjects Mturk Recruited 6 subjects from the lab Inter-annotator Agreement

28. Results

30. Discussion R_GEN and R_SYN performance is comparable to human-generated questions R_NMOD (which …?) outperforms all other question types including human-generated questions R_START rule did not work

31. Key Ideas Use Targeted Clarifications Address challenges with targeted clarifications  Experiment on automatic detection of inappropriate questions  Experiment on automatic detection of when to terminate clarification Data collection for system evaluation

32. Image Description and Questioning Speaker1:  A car is burning behind the girl  The girl looks startled  There was a massive explosion Speaker2:  A woman is standing in front of a burning car  Everything around her seems to have been destroyed  What caused this destruction? Show user an image and ask to describe it and construct questions

33. Data Collection for System Evaluation Advantages: Do not prime users with words in a verbally described scenario Elicits natural speech compared to reading Can be extended to a 2-way dialogue where the interviewee is given a narrative or video information for answering interviewer's questions. Disadvantages: Uncontrolled vocabulary (can not force to mispronounce words) No control across subject pairs

34. Impact Impact on Speech-to-Speech Translation Detecting when a targeted clarification question was inappropriate is an important feature for determining next dialogue move in clarification Impact beyond Speech-to-Speech Translation Targeted clarifications can be used in spoken dialogue systems Especially useful for non-slot-filling (tutoring, virtual assistants)

35. Future Work Appropriate and inappropriate questions Analyze the data collected in responses to appropriate and inappropriate clarification questions Use machine learning to predict if an utterance is an answer to appropriate or inappropriate clarification question Targeted (reprise) clarification questions Which information from an initial sentence should a reprise clarification question contain? Using human-constructed questions, determine which information is essential to be repeated in a targeted question Clarification length How long should the system focus on a targeted clarification before back off? Collect data and use machine learning to predict on each system’s turn whether a clarification should continue or stops

36. Conclusions Used an error-simulation system to collect data  Data collection experiment for automatic detection of answers to 'inappropriate' system clarifications  Evaluation of automatically generated reprise clarification questions shows that they could be used in a system  Proposed an experiment for determining an optimal length of targeted clarification Collected audio data for system evaluation using an image description method 36

37. Thank you Questions? 37

38. Challenge 3: Clarification Length How long should the system focus on a targeted clarification before back off? In a Speech-to-Speech translation: back-off= translate In spoken dialogue systems : back-off = ask a generic question to 'please rephrase'. The answer depends on how patient and cooperative are users.

39. Evaluation of Clarification Length BOLT 2012 system behaviour: System asks targeted clarification at most 3 times before translating. Goal: Determine dynamically at each clarification turn whether the system should terminate clarification process. Use data to learn the dialogue strategy

40. Experiment Design Simulate sequence of unsuccessful clarification questions. Give user an option to hit “translate” button Distractor cases: Simulate successful clarification  User: This computer is not operational  System: Please rephrase “not operational”  User: not working  System: thank you ( translate and show next question) Experimental case:  Loop asking 3 – 5 different targeted questions  Clarification dialogue continues until the user hits “translate” Use a combination of distractor and experimental cases

41. Method Use data to determine when system should give up on a targeted clarification  Apply machine learning  Features: Dialogue length (more likely to give up as dialogue continues to fail) Question type Appropriateness of a clarification question Confidences of error detection and classification components