1. Building and Evaluating SDS
Julia Hirschberg
LSA07 353
9/19/2018

2. Today Natural Language Understanding (NLU) in SDS
Meaning representations
Evaluating SDS
9/19/2018

3. The NLU Component What kinds of meanings do we want to capture?
Entities, actions, events, times…
E.g. an airline travel domain
Entities: flights, airlines, cities
Actions: look up, reserve, fly
Modifiers: first, next, latest
Events: departures, arrivals
Temporals: dates, hours
How should we represent these in our system?
What does an SDS need from a meaning representation?
9/19/2018

4. Be able to Answer questions Determine truth Draw inferences
When does the first flight from Boston to Philadelphia leave on July 7?
Determine truth
Is there a flight from Boston to Philadelphia?
Draw inferences
Is there a connecting flight to Denver?
9/19/2018

5. Most SDS use Semantic Grammars
Robust parsing, or, information extraction
Define grammars specifically in terms of the semantic information we want to extract
Domain specific rules correspond to items in the domain
I want to go from Boston to Baltimore on Thursday, September 24th
TripRequest  Greeting Need-specification travel-verb from City to City on Date
Greeting  {Hello|Hi|Um|ε}
Need-specification  {I want|I need} …
9/19/2018

6. Grammar elements can be mapped directly to slots in semantic frames
From-City
To-City
Date
Airline
Boston
Baltimore
9/24/07 ?
9/19/2018

7. Drawbacks of Semantic Grammars
Lack of generality
A new one for each application
Very costly in development time
Constant updates
Can be very large, depending on how much coverage you want them to have
If users go outside grammar, things may break disastrously
I want to leave from my house at 10 a.m.
I want to talk to a person.
9/19/2018

8. Semantic HMMs
Train sequential model on labeled corpora to identify semantic slots from user input
I want /Need spec to go /Travel verb to Boston /To-city on Thursday /Date.
Relies on training corpus to provide
Likely sequences of semantic slots
Likely words associated with those slots
Can be used to fill slots in template
Or to directly generate a response while processing input
9/19/2018

9. Managing User Input
Rely heavily on knowledge of task and constraints on user actions
Handle fairly sophisticated phenomena
I want to go to Boston next Thursday.
I want to leave from there on Friday for Baltimore.
TripRequest  Need-spec travel-verb from City on Date for City
Dialogue postulates and simple discourse history handle reference resolution
Use prior probabilities based on training corpus to resolve ambiguities, e.g.
Boston Baltimore on July 12
9/19/2018

10. Priming User Input
Hypothesis: Users tend to use the vocabulary and syntax the system uses
Lexical entrainment experiments (Clark & Brennan ’96)
Re-use of system prompt vocabulary/syntax:
Please tell me where you would like to leave/depart from.
I would like to leave/depart from Boston…
Evidence from KTH data collections in field
9/19/2018

11. User Responses to Vaxholm
The answers to the question:
“What weekday do you want to go?”
(Vilken veckodag vill du åka?)
22% Friday (fredag)
11% I want to go on Friday (jag vill åka på fredag)
11% I want to go today (jag vill åka idag)
7% on Friday (på fredag)
6% I want to go a Friday (jag vill åka en fredag)
- are there any hotels in Vaxholm? (finns det några hotell i Vaxholm)
9/19/2018

12. Verb Priming: How often do you go abroad on holiday?
Hur ofta åker du utomlands på semestern?
Hur ofta reser du utomlands på semestern?
jag åker en gång om året kanske
jag åker ganska sällan utomlands på semester
jag åker nästan alltid utomlands under min semester
jag åker ungefär 2 gånger per år utomlands på semester
jag åker utomlands nästan varje år
jag åker utomlands på semestern varje år
jag åker utomlands ungefär en gång om året
jag är nästan aldrig utomlands
en eller två gånger om året
en gång per semester
kanske en gång per år
ungefär en gång per år
åtminståne en gång om året
nästan aldrig
jag reser en gång om året utomlands
jag reser inte ofta utomlands på semester det blir mera i arbetet
jag reser reser utomlands på semestern vartannat år
jag reser utomlands en gång per semester
jag reser utomlands på semester ungefär en gång per år
jag brukar resa utomlands på semestern åtminståne en gång i året
en gång per år kanske
en gång vart annat år
varje år
vart tredje år ungefär
nu för tiden inte så ofta
varje år brukar jag åka utomlands
9/19/2018

13. Results no reuse no answer 4% 2% other 24% reuse 52% 18% ellipse
9/19/2018

14. Issue: Training and SDS
Is implicit training ‘better’ than explicit?
Is it easier to train the user than to retrain the system
9/19/2018

15. Evaluating Dialogue Systems
PARADISE framework (Walker et al ’00)
“Performance” of a dialogue system is affected both by what gets accomplished by the user and the dialogue agent and how it gets accomplished
Maximize
Task Success
Minimize
Costs
Efficiency
Measures
Qualitative
Measures
9/19/2018

16. Task Success Task goals seen as Attribute-Value Matrix
ELVIS retrieval task (Walker et al ‘97)
“Find the time and place of your meeting with Kim.”
Attribute Value
Selection Criterion Kim or Meeting
Time 10:30 a.m.
Place 2D516
Task success defined by match between AVM values at end of with “true” values for AVM
9/19/2018

17. Metrics
Efficiency of the Interaction:User Turns, System Turns, Elapsed Time
Quality of the Interaction: ASR rejections, Time Out Prompts, Help Requests, Barge-Ins, Mean Recognition Score (concept accuracy), Cancellation Requests
User Satisfaction
Task Success: perceived completion, information extracted
9/19/2018

18. Experimental Procedures
Subjects given specified tasks
Spoken dialogues recorded
Cost factors, states, dialog acts automatically logged; ASR accuracy,barge-in hand-labeled
Users specify task solution via web page
Users complete User Satisfaction surveys
Use multiple linear regression to model User Satisfaction as a function of Task Success and Costs; test for significant predictive factors
9/19/2018

19. User Satisfaction: Sum of Many Measures
Was Annie easy to understand in this conversation? (TTS Performance)
In this conversation, did Annie understand what you said? (ASR Performance)
In this conversation, was it easy to find the message you wanted? (Task Ease)
Was the pace of interaction with Annie appropriate in this conversation? (Interaction Pace)
In this conversation, did you know what you could say at each point of the dialog?
(User Expertise)
How often was Annie sluggish and slow to reply to you in this conversation? (System Response)
Did Annie work the way you expected her to in this conversation? (Expected Behavior)
From your current experience with using Annie to get your , do you think you'd use Annie regularly to access your mail when you are away from your desk? (Future Use)
9/19/2018

20. Performance Functions from Three Systems
ELVIS User Sat.= .21* COMP * MRS * ET
TOOT User Sat.= .35* COMP + .45* MRS - .14*ET
ANNIE User Sat.= .33*COMP + .25* MRS +.33* Help
COMP: User perception of task completion (task success)
MRS: Mean recognition accuracy (cost)
ET: Elapsed time (cost)
Help: Help requests (cost)
9/19/2018

21. Performance Model
Perceived task completion and mean recognition score are consistently significant predictors of User Satisfaction
Performance model useful for system development
Making predictions about system modifications
Distinguishing ‘good’ dialogues from ‘bad’ dialogues
But can we also tell on-line when a dialogue is ‘going wrong’
9/19/2018

22. Next Class
J&M 22.5
Jurafsky et al ’98
Rosset & Lamel ‘04
9/19/2018