1. A Framework For Developing Conversational User Interfaces
James Glass, Eugene Weinstein, Scott Cyphers, Joseph Polifroni
MIT Computer Science and Artificial Intelligence Laboratory Cambridge, MA USA
Grace Chung
Corporation for National Research Initiatives
Reston, VA USA
Mikio Nakano
NTT Corporation
Atsugi, Japan

2. Conversational User Interfaces
Coding deals with:
bit rate reduction
high quality
transformation
Synthesis: generation of intelligible speech of high quality from unrestricted text
Both fields are much further advanced than ASR
What is an ASR system?
Conversational User Interfaces
Speech
Speech
Synthesis
Recognition
Text
Human
Computer
Generation
Text
Understanding
Meaning
Eugene Weinstein — MIT Computer Science and Artificial Intelligence Laboratory
Computer Aided Design on User Interfaces – Jan 16th, 2004

3. Types of Conversational Interfaces
Conversational systems differ in the degree with which human or computer controls the conversation (initiative)
Human
Computer
Initiative
Computer maintains tight control
Human is highly restricted
C: Please say the departure city.
Human takes complete control
Computer is totally passive
H: I want to visit my grandmother.
Directed
Dialogue
Mixed Initiative
Dialogue
Free Form
Dialogue
Eugene Weinstein — MIT Computer Science and Artificial Intelligence Laboratory
Computer Aided Design on User Interfaces – Jan 16th, 2004

4. Conversational Interfaces
Can understand verbal input
Speech recognition
Language understanding (in context)
Language
Generation
Can engage in dialogue with a user during the interaction
Speech
Synthesis
Dialogue
Management
Can verbalize response
Language generation
Speech synthesis
Audio
Back End
Speech
Recognition
Context
Resolution
Language Understanding
Eugene Weinstein — MIT Computer Science and Artificial Intelligence Laboratory
Computer Aided Design on User Interfaces – Jan 16th, 2004

5. The Problem With Conversational Interfaces
Advanced conversational systems are out there
Both user and computer can take initiative
Goal: conversational skill of system should approach that of human operator
But…
These systems are built by experts
Huge learning curve for novices, and
Tremendous iterative effort required even from experts
For this reason
Most advanced conversational systems remain in research labs
e.g. Jupiter weather info system ( TALK) : Zue et al, IEEE Trans. SAP, 8(1), 2000
However, we have seen limited commercial deployment
e.g. AT&T’s “How May I Help You”, Gorin et al, Speech Communication, 23, 1997
Eugene Weinstein — MIT Computer Science and Artificial Intelligence Laboratory
Computer Aided Design on User Interfaces – Jan 16th, 2004

6. Simplifying Conversational System Creation
Goal: make it easier for both expert and novice developers to create conversational interfaces
But still use advanced human language technologies
Strategy: simplify configuration process
Automatically configure technology components bases on examples
Allow specification through web interface or unified configuration file
SpeechBuilder
Configuration
Engine
Web
Interface
Configuration
File
Context
Resolution
Dialogue
Management
Generation
Synthesis
Understanding
Recognition
Eugene Weinstein — MIT Computer Science and Artificial Intelligence Laboratory
Computer Aided Design on User Interfaces – Jan 16th, 2004

7. Configuring a Conversational Interface: Knowledge Representation
First, define example sentences for in-domain actions
Action
Examples
identify
I would like to know today’s weather in Denver
What will the temperature be on Tuesday
set
Turn on the radio in the kitchen please
Can you turn the dining room lights off
Then, define the important concepts present in the actions (attributes):
Concept values make up recognizer vocabulary!
Examples of attributes automatically matched to attribute classes
Attribute
Values
city
Boston, Denver, San Francisco, …
room
living room, dining room, kitchen, …
Eugene Weinstein — MIT Computer Science and Artificial Intelligence Laboratory
Computer Aided Design on User Interfaces – Jan 16th, 2004

8. Starting with a Database Table
Provide database table to configure speech interface:
Name
Phone
Office
Jim Glass
x3-1640
601
Scott Cyphers
x3-0248
604
Eugene Weinstein
X3-8569
633
Only some columns are used to access entries (e.g., Name)
Values of those columns become values for domain concepts
Default action sentences are automatically generated
But, every table cell can potentially be an answer to a question
All Names of columns become one concept – “property”
Attributes
name
Jim Glass, Scott Cyphers…
property
Name, Phone, , Office
Actions
request_property
What is the for Jim Glass?
request_office
Where can I find Jim Glass?
Eugene Weinstein — MIT Computer Science and Artificial Intelligence Laboratory
Computer Aided Design on User Interfaces – Jan 16th, 2004

9. Generic Dialogue Manager Language Understanding
Dialogue Management
Generic Dialogue Manager (Polifroni & Chung, ICSLP 2002)
Language
Generation
Hotels
Generic Dialogue Manager
Dialogue
Management
Air Travel
Speech
Synthesis
Sports
Weather
Audio
Back End
Plan system responses
Regularize common concepts
Summarize database results
Speech
Recognition
Context
Resolution
Language Understanding
Eugene Weinstein — MIT Computer Science and Artificial Intelligence Laboratory
Computer Aided Design on User Interfaces – Jan 16th, 2004

10. Context Resolution “Show me restaurants in Cambridge.” Resolve Deixis
Input Query
“Show me restaurants in Cambridge.”
Resolve Deixis
“What does this one serve?”
Resolve Pronouns
“What is their phone number?”
Inherit Predicates
“Are there any on Main Street?”
Incorporate Fragments
“What about Massachusetts Ave?”
Fill in Default Values
“Give me directions from MIT.”
Query Interpreted in Context
Eugene Weinstein — MIT Computer Science and Artificial Intelligence Laboratory
Computer Aided Design on User Interfaces – Jan 16th, 2004

11. Human Language Technology Details
Approach: Use same technologies as deployed in our mainstream, more complex systems
Speech Recognizer (Glass, Computer, Speech, and Language, 2003)
Trained on 100+ hours of mostly telephone speech
Word pronunciations supplied by large dictionary, generated by rule, or provided by developer
Natural Language Understanding: (Seneff, Computational Linguistics, 1992)
Hierarchical sentence grammar used to parse sentence hypothesis
Back off to concept spotting when no full parse is made
Language Generation: (Baptist&Seneff, ICSLP 2000)
Used in: SQL (DB Query) generation, paraphrasing & URL-encoding meaning representation, responses
Eugene Weinstein — MIT Computer Science and Artificial Intelligence Laboratory
Computer Aided Design on User Interfaces – Jan 16th, 2004

12. Concepts (Attributes)
Web-based Interface
Defining Actions and
Concepts (Attributes)
Eugene Weinstein — MIT Computer Science and Artificial Intelligence Laboratory
Computer Aided Design on User Interfaces – Jan 16th, 2004

13. Web-based Interface: Viewing Sentences
Examining how sentences are reduced
to an action and a set of attribute-value pairs
Eugene Weinstein — MIT Computer Science and Artificial Intelligence Laboratory
Computer Aided Design on User Interfaces – Jan 16th, 2004

14. Web-based Interface: Response Generation
Domain independent system prompts
Customizing system
responses
Domain specific system prompts
Eugene Weinstein — MIT Computer Science and Artificial Intelligence Laboratory
Computer Aided Design on User Interfaces – Jan 16th, 2004

15. Web-based Interface: Editing Pronunciations
Modifying system generated
pronunciations for the vocabulary
Eugene Weinstein — MIT Computer Science and Artificial Intelligence Laboratory
Computer Aided Design on User Interfaces – Jan 16th, 2004

16. Web-based Interface: Context Resolution
Context Resolution configured through
Masking and Inheritance of concepts
Eugene Weinstein — MIT Computer Science and Artificial Intelligence Laboratory
Computer Aided Design on User Interfaces – Jan 16th, 2004

17. Voice Configuration File: An Alternative to the Web Interface
Entire domain can be specified in single configuration file
Allows for automated generation of conversational systems
<actions>
<request_name> = i would like a restaurant
| can you (show|give) me a Chinese restaurant in Arlington;
</actions>
<attributes>
<cuisine> = Chinese|Taiwanese;
<city> = Washington | Boston | Arlington;
</attributes>
<discourse>
name masks(city cuisine neighborhood);
</discourse>
<constraints>
<request_name> (city|neighborhood) {prompt_for_city};
</constraints>
Eugene Weinstein — MIT Computer Science and Artificial Intelligence Laboratory
Computer Aided Design on User Interfaces – Jan 16th, 2004

18. Computer Aided Design on User Interfaces – Jan 16th, 2004
Deployment
SpeechBuilder functional for the past three years
Some example domains:
Office appliance control
Laboratory directory (auto-attendant)
Restaurant query system
Has been used by MIT researchers (experts) as well as novice developers at our sponsor companies
Used in technology transfer workshop for pervasive computing project (Oxygen)
SpeechBuilder has been used as an educational tool
Computational linguistics class at Georgetown University
Summer class at Johns Hopkins University
Youngest SpeechBuilder developer: 9 years old
Eugene Weinstein — MIT Computer Science and Artificial Intelligence Laboratory
Computer Aided Design on User Interfaces – Jan 16th, 2004

19. Japanese SpeechBuilder
Created in collaboration with NTT
Challenge: Segmentation (no spaces between words)
Eugene Weinstein — MIT Computer Science and Artificial Intelligence Laboratory
Computer Aided Design on User Interfaces – Jan 16th, 2004

20. Computer Aided Design on User Interfaces – Jan 16th, 2004
Example Domain
A hotel application using the generic dialogue manager
Compiled via SpeechBuilder using constraints shown previously
Other generic functionality is automatically included
Illustrated technical issues:
Soliciting necessary information from user
Interpreting fragments correctly in context
Canonicalizing relative dates
Ordering and summarizing results of query to content provider
Resolving superlatives/updating discourse context
Interpreting pronouns in context
Returning and speaking specific properties
Repeating previous replies
Eugene Weinstein — MIT Computer Science and Artificial Intelligence Laboratory
Computer Aided Design on User Interfaces – Jan 16th, 2004

21. Another Example Domain: Object Manipulation System
Stock SpeechBuilder domain for spoken dialogue
Custom back-end connected to stereo camera and person tracking algorithm (Demirdjian, WOMOT 2003)
Eugene Weinstein — MIT Computer Science and Artificial Intelligence Laboratory
Computer Aided Design on User Interfaces – Jan 16th, 2004

22. Ongoing and Future Work
Incorporate speech synthesis
Allow use of concatenative speech synthesizer (Yi et al, ICSLP 2000) in SpeechBuilder
Allow use of multiple modalities
Provide functionality to incorporate multimodal input into systems
Improve dialogue management tools and modules
Improve ability of SpeechBuilder systems to use more sophisticated dialogue strategies
Provide additional generic semantic concepts for use in domains
Allow system refinement by unsupervised learning
Use confidence scores to improve domain language model (Nakano&Hazen, Eurospeech 2003)
Allow system modification in real-time
Need ability to re-train recognizer during runtime (Schalkwyk et al, Eurospeech 2003)
Eugene Weinstein — MIT Computer Science and Artificial Intelligence Laboratory
Computer Aided Design on User Interfaces – Jan 16th, 2004

23. Thank You! For more information:
us!
Jupiter weather Information system:
(outside USA)
TALK (USA toll-free)
Mercury flight information system:
(outside USA)
1-877-MIT-TALK (USA toll-free)
Pegasus flight status system:
(outside USA)
1-877-LCS-TALK (USA toll-free)
Eugene Weinstein — MIT Computer Science and Artificial Intelligence Laboratory
Computer Aided Design on User Interfaces – Jan 16th, 2004

24. Computer Aided Design on User Interfaces – Jan 16th, 2004
THE END
Eugene Weinstein — MIT Computer Science and Artificial Intelligence Laboratory
Computer Aided Design on User Interfaces – Jan 16th, 2004

25. Computer Aided Design on User Interfaces – Jan 16th, 2004
Utility for rapid prototyping of speech-based interfaces
Used to create demonstrations for NTT CS Labs open house
Prototypes were developed with a few days of effort
Three papers submitted for publishing
Eugene Weinstein — MIT Computer Science and Artificial Intelligence Laboratory
Computer Aided Design on User Interfaces – Jan 16th, 2004

26. Computer Aided Design on User Interfaces – Jan 16th, 2004
Human Language
Technologies
Only some columns are used to access entries (e.g., Name)
Values of those columns become values for domain concepts
Default action sentences are automatically generated
But, every table cell can potentially be an answer to a question
Names of non-access columns become a concept
Eugene Weinstein — MIT Computer Science and Artificial Intelligence Laboratory
Computer Aided Design on User Interfaces – Jan 16th, 2004

27. To Configure Response Generation…
For each concept present in the domain, define how queries about that concept should be answered
<telephone> = “The telephone for :name is :phone”
Define some prompts for generic events, e.g. welcome and goodbye
<welcome> = “Welcome to the auto-attendant”
<no_data> = “Sorry, there was no data matching your request.”
Eugene Weinstein — MIT Computer Science and Artificial Intelligence Laboratory
Computer Aided Design on User Interfaces – Jan 16th, 2004

28. Conversational User Interfaces: Input Side
Human Language
Technologies
Speech
Text
Recognition
“Find me a flight to
Boston on Tuesday”
Meaning
Understanding
“Back-end”
Technologies
action=flights
to_city=Boston
day=Tuesday
Action DB
Eugene Weinstein — MIT Computer Science and Artificial Intelligence Laboratory
Computer Aided Design on User Interfaces – Jan 16th, 2004

29. Conversational User Interfaces: Output Side
Human Language
Technologies
Speech
Synthesis
Delta flight, number fifty five
from La Guardia to Boston…
Text
Generation
flight_num=55
airline=Delta
origin=LGA
dest=BOS
Meaning DB
Action
Eugene Weinstein — MIT Computer Science and Artificial Intelligence Laboratory
Computer Aided Design on User Interfaces – Jan 16th, 2004

30. Conversational User Interfaces: The Whole Picture
Or Is It?
Speech
Speech
Text
Recognition
Synthesis
Text
Meaning
Understanding
Generation
Meaning
Action
Eugene Weinstein — MIT Computer Science and Artificial Intelligence Laboratory
Computer Aided Design on User Interfaces – Jan 16th, 2004

31. The Missing Pieces: Context and Dialogue
Context Resolution:
action=flights
to_city=Boston
day=Tuesday
Last time,
the user asked
for a flight
from LGA
action=flights
origin=BOS
dest=LGA
day=Tuesday + =
Dialogue Management:
action=flights
to_city=Boston
day=Tuesday
“Which city
would you like
to fly from?” + =
Eugene Weinstein — MIT Computer Science and Artificial Intelligence Laboratory
Computer Aided Design on User Interfaces – Jan 16th, 2004

32. Conversational User Interfaces: The Whole Picture
Speech
Speech
Text
Recognition
Synthesis
Text
Understanding
Generation
Meaning
Meaning
Context Resolution,
Dialogue Management
Action
Eugene Weinstein — MIT Computer Science and Artificial Intelligence Laboratory
Computer Aided Design on User Interfaces – Jan 16th, 2004

33. The Problem With Conversational Interfaces…
Complex conversational systems are out there
Both user and computer can take initiative
Goal: conversational skill of system should approach that of human operator
But…
These systems are built by experts
Huge learning curve for novices, and
Tremendous iterative effort required even from experts
For this reason
Most advanced conversational systems remain in research labs
e.g. Jupiter weather info system ( TALK) : Zue et al, IEEE Trans. SAP, 8(1), 2000
However, we have seen limited commercial deployment
e.g. AT&T’s “How May I Help You”, Gorin et al, Speech Communication, 23, 1997
Eugene Weinstein — MIT Computer Science and Artificial Intelligence Laboratory
Computer Aided Design on User Interfaces – Jan 16th, 2004

34. Configuring Response Generation…
For each concept present in the domain, define how queries about that concept should be answered
Configure some generic prompts for summarizing long results
Define some prompts for generic events, e.g. welcome
Property/ Condition
Response
phone
The phone number for :restaurant_name is :phone
cuisine
:restaurant_name serves :cuisine cuisine
Welcome
Welcome to the restaurants domain
No matches
I’m sorry, I couldn’t find any restaurants matching your request
Many matches
I found five restauraunts :items
item (what to return when summarizing)
:restaurant_name
Eugene Weinstein — MIT Computer Science and Artificial Intelligence Laboratory
Computer Aided Design on User Interfaces – Jan 16th, 2004

35. Configuring Context Resolution
Context Resolution (discourse) configured through Masking and Inhertiance of concepts
Inheritance configures how actions remember concepts, e.g.:
User: “What is the phone number for Jim Glass”
System: “Jim Glass’ phone number is
User: “What about his address?”
System: “Jim Glass’ address is
Name concept is inherited
Masking configures how certain concepts block other concepts, even in the presence of inheritance, e.g.
User: “Do you have any restaurants in Boston?”
System: “In Boston, I have the following…”
User: “What about in Times Square?”
System: “In Times Square, New York, I have…”
City concept is masked by Neighborhood concept
Name is inherited
City is masked
Eugene Weinstein — MIT Computer Science and Artificial Intelligence Laboratory
Computer Aided Design on User Interfaces – Jan 16th, 2004

36. Voice Configuration File
Developers can also use Voice Configuration (VCFG) file format to configure SpechBuilder domains:
<actions>
<request_name> = i would like a restaurant
| can you (show|give) me a Chinese restaurant in Arlington;
</actions>
<attributes>
<cuisine> = Chinese|Taiwanese;
<city> = Washington | Boston | Arlington;
</attributes>
<discourse>
name masks(city cuisine neighborhood);
</discourse>
<constraints>
<request_name> (city|neighborhood) {prompt_for_city};
</constraints>
Eugene Weinstein — MIT Computer Science and Artificial Intelligence Laboratory
Computer Aided Design on User Interfaces – Jan 16th, 2004

37. Generic Dialogue Manager Language Understanding
Dialogue Management
Generic Dialogue Manager (Polifroni & Chung, ICSLP 2002)
Hotels
Language
Generation
Generic Dialogue Manager
Air Travel
Speech
Synthesis
Sports
Dialogue
Management
Weather
Plan system responses
Regularize common concepts
Summarize database results
Audio
Database
Speech
Recognition
Context
Resolution
Language Understanding
Eugene Weinstein — MIT Computer Science and Artificial Intelligence Laboratory
Computer Aided Design on User Interfaces – Jan 16th, 2004

38. Computer Aided Design on User Interfaces – Jan 16th, 2004
Deployment
SpeechBuilder functional for the past three years
Some example domains:
Office appliance control
Laboratory directory (auto-attendant)
Restaurant query system
Has been used by MIT researchers (experts) as well as novice developers at our partner companies
SpeechBuilder has been used by students in
Computational linguistics class at Georgetown University
Summer class at Johns Hopkins University
Technology transfer workshop for pervasive computing project (Oxygen)
In collaboration with NTT, we have developed a Japanese version of SpeechBuilder. Japanese domains:
Bus timetable system
Weather information system
Eugene Weinstein — MIT Computer Science and Artificial Intelligence Laboratory
Computer Aided Design on User Interfaces – Jan 16th, 2004

39. Configuring a Speech Interface with SpeechBuilder: Knowledge Representation
First define some concepts present in the domain (attributes):
Concept values make up recognizer vocabulary!
Attribute
Values
city
Boston, Denver, San Francisco, …
room
living room, dining room, kitchen, …
Then, define examples of things to do with the concepts (actions)
Examples of attributes automatically matched to attribute classes
Action
Examples
identify
I would like to know today’s weather in Denver
What will the temperature be on Tuesday
set
Turn on the radio in the kitchen please
Can you turn the dining room lights off
Eugene Weinstein — MIT Computer Science and Artificial Intelligence Laboratory
Computer Aided Design on User Interfaces – Jan 16th, 2004