1. Issues in Spoken Dialogue Systems
Julia Hirschberg
LSA07 353
9/21/2018

2. Why Spoken Dialogue Systems?
Science
A chance to study and try to emulate human behavior in spoken communication
Technology
Demonstrate utility of several speech technolgies
A means to save human labor and permit flexible management of simple tasks
Online banking services
Reservations (air, train, bus, rental car)
Information access
9/21/2018

3. The Instructor
Julia Hirschberg
Columbia Speech Lab
9/21/2018

4. Course Overview
Discussion of past and current SDS, with examples of interactions
Major problems in building SDS
Problems in understanding and modeling human conversational behaviors
Limits of the available technologies
Problems in evaluating SDS
SDS of the future
9/21/2018

5. Some Early SDS Projects at KTH
The Waxholm Project
Kiosk provides information on travel in the Stockholm archipelago
August
Kiosk provides information on August Strindberg
Other demos at
9/21/2018

6. The Waxholm system When do the evening boats depart?
I think I want to go to Waxholm
The city
This is a table of the boats...
Information about the restaurants in Waxholm
is shown in this table
Is it possible to eat in Waxholm?
Which day of the week do you want to go?
I want to go tomorrow
There are lots of boats from Stockholm to Waxholm
on a Friday, At what time do you want to go?
I am looking for boats to Waxholm
From where do you want to go
Thank you
Information about the hotels in Waxholm is shown in this table
Thank you too
Where can I find hotels?
Waxholm is shown on this map
Information about hotels is shown in this table
Which hotels are in Waxholm?
Where is Waxholm?
9/21/2018

7. Issues Limited domain: the more so the better
Use of graphical display (maps, tables, charts) to present complex information and enhance visual appeal
Types of initiative in SDS
System
User
Mixed
Importance of lexical priming
9/21/2018

8. The August system Strindberg was married three times!
Yes, it might be that we will!
What is your name?
Strindberg was born in 1849
Over a million people live
in the Stockholm area
What do you do for a living?
I call myself Strindberg, but I
don’t really have a surname
I can answer questions about
Strindberg, KTH and Stockholm
When were you born?
Do you like it here?
How many people live in
Stockholm?
The information is shown
on the map
People who live in glass houses
should not throw stones
I come from the department
of Speech, Music and Hearing
Yes, that was a smart thing
to say!
Good bye!
Thank you!
The Royal Institute of
Technology!
You are welcome!
Perhaps we will meet
soon again!
9/21/2018

9. Issues More open domain: entertaining vs. accomplishing a real task
Use of multimodal (speech and face) synthesis
Gesture (face)
System ‘personality’
A good tool for data collection
How to evaluate?
9/21/2018

10. More Recent AT&T Experimental Work
Toot system created to study differences in types of initiative and confirmation strategies and identify user preferences ’98-99
Some successful dialogues
Some not so successful
9/21/2018

11. Issues Automatic Speech Recognition (ASR) errors
Importance of age, gender, accent
Text-to-Speech (TTS) issues
Error detection and correction
Hyperarticulation
Confirmation strategies
Explicit
Implicit
None
Managing turn-taking
9/21/2018

12. Evaluation again….how do we know when a change has been beneficial?
9/21/2018

13. A ‘Real’ SDS System from the Lab…
Let’s Go from CMU
What type of initiative supported?
What type of confirmation stragegy?
Difficult or easy to evaluate?
Compare to the AT&T system? The KTH?
How natural is the interaction?
What are they doing to enhance, over the other systems?
9/21/2018

14. The Course Classes, readings, and assignments
Modeling Turn-taking Behaviors
Components of SDS
Automatic Speech Recognition (ASR)
Text-to-Speech (TTS)
Natural Language Understanding (NLU)
Building and Evaluating SDS
Interpreting and Generating Dialogue Acts
Error Detection and Repair Strategies
Making SDS More Human: Entrainment/Adaptation
Can systems become more like their users?
Should they?
9/21/2018

15. The Course Paper
Eliza is a classic text-based AI Dialogue System which often fools casual users into thinking they are conversing with a human being. For this assignment:
Become acquainted with the text Eliza:
Identifying 5 or more strategies you can use to demonstrate that Eliza is not a human conversationalist (e.g. linguistic constructions she does not handle well, pragmatic behaviors she does not generate appropriately or recognize). As evidence describe your inputs and her outputs
Next, design a speech-enabled version of Eliza:
Explain the difficulties you will have to overcome in recognition, generation, and dialogue management, compared to the text version, based upon the topics we have studied in class and additional observations you may have
Suggest specific ways of dealing with these problems based upon what you know of the state of the art in SDS or ideas you may have to improve upon it
Indicate the ways in which a spoken version of Eliza might be even better than the text version, in terms of how it might interact with the user and the features of spoken language you might be able to make use of in generation and recognition of user behavior
9/21/2018

16. SDS and Popular Culture: SNL
Amtrak’s Julie
Next class: J&M 22.1, Clark ’03, Beattie ‘82
9/21/2018